Regular expression generation based on positive and negative pattern matching examples

ABSTRACT

Disclosed herein are techniques related to automated generation of regular expressions. In some embodiments, a regular expression generator may receive input data comprising one or more character sequences. The regular expression generator may convert character sequences into a sets of regular expression codes and/or span data structures. The regular expression generator may identify a longest common subsequence shared by the sets of regular expression codes and/or spans, and may generate a regular expression based upon the longest common subsequence.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Patent Application No. 62/684,498, filed Jun. 13, 2018,entitled “AUTOMATED GENERATION OF REGULAR EXPRESSIONS,” and also claimspriority under 35 U.S.C. § 119(e) to U.S. Provisional Patent ApplicationNo. 62/749,001, filed Oct. 22, 2018, entitled “AUTOMATED GENERATION OFREGULAR EXPRESSIONS.” The entire contents of U.S.

Provisional Patent Application Nos. 62/684,498 and 62/749,001 areincorporated herein by reference for all purposes.

BACKGROUND

Big data analytics systems can be used for predictive analytics, userbehavior analytics, and other advanced data analytics. However, beforeany data analysis may be performed effectively to provide usefulresults, the initial data set may need to be formatted into clean andcurated data sets. This data onboarding often presents challenges forcloud-based data repositories and other big data systems, where datafrom various different data sources and/or data streams may be compiledinto a single data repository. Such data may include structured data inmultiple different formats, semi-structured data in accordance withdifferent data models, and even unstructured data. Repositories of suchdata often include data representations within various different formatsand structures, and also may include duplicate data and erroneous data.When these data repositories are analyzed for reporting, predictivemodeling, and other analytics tasks, a poor signal-to-noise ratio of theinitial data set may lead to results that are inaccurate or not useful.

Many current solutions to the problems of data formatting andpreprocessing include manual and ad hoc processes to clean and curatethe data, in order to manipulate the data into a common format beforeperforming a data analysis. While these manual processes can beeffective for certain smaller data sets, such processes may beinefficient and impractical when attempting to preprocess and formatlarge-scale data sets.

BRIEF SUMMARY

Aspects described herein provide various techniques for generatingregular expressions. As used herein, a “regular expression” may refer toa sequence of characters defining a pattern, which may be used to searchfor matches within longer input text strings. In some embodiments,regular expressions may be composed using a symbolic wildcard-matchinglanguage, and the patterns defined by regular expressions may be used tomatch character strings and/or extract information from characterstrings provided as input. In various embodiments described herein, aregular expression generator implemented as data processing system maybe used to receive and display input text data, receive selections via aclient user interface of specific character subsets of the input text,and then generate one or more regular expressions based on the selectedcharacter subsets. After generating one or more regular expressions, aregular expression engine may be used to match the pattern of theregular expression against one or more data sets. In variousembodiments, data matching the regular expression may be extracted,reformatted, or modified, etc. In some cases, additional columns,tables, or other data sets may be created based on the data matching theregular expression.

According to certain aspects described herein, a regular expressiongenerator implemented via a data processing system may generate regularexpressions based upon a determined longest common subsequence (LCS)that is shared by different sets of one or more regular expressioncodes. Regular expression codes (which also may be referred to ascategory codes) may include, for example, L for letters of the Englishalphabet, N for numbers, Z for white spaces, P for punctuation marks,and S for other symbols. Each set of one or more regular expressioncodes may be converted from a different sequence of one or morecharacters received as input data through a user interface. Regularexpression codes excluded from the LCS may be represented as optionaland/or alternatives. In some embodiments, a regular expression code maybe associated with a minimum number of occurrences of the regularexpression code. Additionally or alternatively, the regular expressioncode may be associated with a maximum number of occurrences of theregular expression code. For example, a set of category codes maycomprise L<0,1> to indicate that a particular portion of an LCS includesa letter at most once if at all. As discussed in more detail below,generalizing the input data as intermediate regular expression codes(IRECs) may provide various technical advantages, including, using verylittle input data, enabling near-instantaneous generation of regularexpressions that do not succumb to false positive matches or falsenegative matches in yet-to-be-seen data.

According to additional aspects described herein, a regular expressionmay be generated based on input data comprising three or more charactersequences. When three or more character sequences are identified asinput data, a regular expression generator that identifies the LCS ofthe character sequences may result in an exponential increase inruntime. In order to identify the LCS of all character sequences in aperformant manner, the regular expression generator may perform an LCSalgorithm on each distinct combination of two character sequences. Afully-connected graph may be generated based on the results of the LCSalgorithms, where each graph node represents a different charactersequence and the length of each graph edge corresponds to the LCS of thenodes defining the graph edge. The order for selecting charactersequences then may be determined by performing a depth-first traversalof a minimum spanning tree for the fully-connected graph.

Further aspects described herein relate to generating regularexpressions based on input including both positive character sequenceexamples and negative character sequence examples. A positive examplemay refer to sequence of characters that are to match the regularexpression to be generated, while a negative example may refer to asequence of characters that are not to match the regular expression tobe generated. In some embodiments, when both positive and negativeexamples are received, the regular expression generator may identify adiscriminator, or shortest subsequence of one or more characters thatdistinguish the positive example(s) from the negative example(s). Theselected discriminator may be a shortest sequence (e.g., expressed incategory codes), and may either be positive or negative, so that thepositive examples will match and the negative examples will not. Thediscriminator then may be hard-coded into the regular expression that isgenerated by the regular expression generator. In some cases, theshortest subsequence may be included in a prefix or suffix portion ofthe negative example(s).

Additional aspects described herein relate to one or more userinterfaces through which input data may be provided to generate regularexpressions. In some embodiments, a user interface may be displayed at aclient device communicatively coupled to the regular expressiongenerator server. The user interface may be generated programmaticallyby the server, by the client device, or by a combination of softwarecomponents executing at the server and the client. Input data receivedvia the user interface may correspond to user selections of one or morecharacter sequences, which may represent positive or negative examples.In some cases, the user interface may support input data that includes aselection of a first character sequence within a second charactersequence. For instance, a user may highlight one or more characterswithin a larger previously highlighted character sequence, and thesecond user selection may provide context for the larger first userselection. This enables input data to be provided to the regularexpression generator with greater specificity, and to provide theregular expression generator with “context” so that it can generateregular expressions that avoid false positives. In response to a userselection of a character sequence via the user interface, the regularexpression generator may generate and display a regular expression. Forexample, when a user highlights a first sequence of characters, theregular expression generator may generate and display a regularexpression matching the first sequence of characters, as well as othersimilar character sequences (e.g., aligning with the intentions of theuser for matching sequences). When the user highlights a second sequenceof characters, the regular expression generator may generate an updatedregular expression which encompasses both the first and second sequencesof characters. Then, when the user highlights a third sequence ofcharacters (e.g., within either the first or second sequence) theregular expression generator may update the regular expression again,and so on.

In accordance with additional aspects described herein, regularexpressions may be generated based on the longest common subsequencefrom one or more input sequence examples, but also may handle charactersthat are present in only some of the examples. To handle characters thatare present in only some input examples, spans may be defined in whichboth a minimum and maximum number of occurrences of a regular expressioncode are tracked. In cases when a span might not present at all of thegiven input examples, the minimum number of occurrences may be set tozero. These minimum and maximum numbers can then be mapped to theregular expression multiplicity syntax. A longest common subsequence(LCS) algorithm may be run on the spans of characters derived from theinput examples, including “optional” spans (e.g., minimum length ofzero) which do not appear in every input example. As discussed below,consecutive spans may be merged during the execution of the LCSalgorithm. In such cases, when extra optional spans that are beingcarried along end up appearing consecutively, the LCS algorithm may berun recursively on those optional spans as well.

Further aspects described herein relate to a combinatoric search, inwhich the LCS algorithm executed by the regular expression generator maybe run multiple times to generate a “correct” regular expression (e.g.,a regular expression that properly matches all given positive examplesand properly excludes all given negative examples), and/or to generatemultiple correct regular expressions from which a most desirable oroptimal regular expression may be selected. In some embodiments, an LCSalgorithm may generally be executed right-to-left on the input examplesto generate a regular expression. However, for comparison purposes andto find alternative regular expressions, the LCS algorithm may beseparately executed backward (e.g., in the left-to-right direction) onthe input examples. For example, the example character sequencesreceived as user input may be reversed before they are run through theLCS algorithm, and the results from the LCS algorithm then may bereversed back (including the original text fragments). Further, in someembodiments, the LCS algorithm may be run multiple times by the regularexpression generator, both in the usual character sequence order and thereverse order, with anchoring at the beginning of the line, anchoring atthe end of the line, and no anchoring at the beginning or end the line.Thus, in some cases, the LCS algorithm may be execute at least these sixtimes, and the shortest successful regular expression may be selectedfrom these executions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating components of an exemplarydistributed system for generating regular expressions, in which variousembodiments may be implemented.

FIG. 2 is a flowchart illustrating a process for generating regularexpressions based on input received via a user interface, according toone or more embodiments described herein.

FIG. 3 is a flowchart illustrating a process for generating regularexpressions using a longest common subsequence (LCS) algorithm on setsof regular expression codes, according to one or more embodimentsdescribed herein.

FIG. 4 is an example diagram for generating a regular expression basedon two character sequence examples, using a longest common subsequence(LCS) algorithm on sets of regular expression codes, according to one ormore embodiments described herein.

FIG. 5 is a flowchart illustrating a process for generating regularexpressions using a longest common subsequence (LCS) algorithm on largersets of regular expression codes, according to one or more embodimentsdescribed herein.

FIG. 6 is an example diagram for generating a regular expression basedon five character sequence examples, using a longest common subsequence(LCS) algorithm on sets of regular expression codes, according to one ormore embodiments described herein.

FIG. 7 is a flowchart illustrating a process for determining an order ofexecution for a longest common subsequence (LCS) algorithm on largersets of regular expression codes, according to one or more embodimentsdescribed herein.

FIGS. 8A and 8B show a fully-connected graph and a minimum spanning treerepresentation of the fully-connected graph, used for determining anorder of execution for a longest common subsequence (LCS) algorithm onlarger sets of regular expression codes, according to one or moreembodiments described herein.

FIG. 9 is a flowchart illustrating a process for generating a regularexpression based on positive and negative character sequence examples,according to one or more embodiments described herein.

FIGS. 10A and 10B are example user interface screens showing generationof regular expressions based on positive and negative character sequenceexamples, according to one or more embodiments described herein.

FIG. 11 is a flowchart illustrating a process for generating regularexpressions based on user data selections received within a userinterface, according to one or more embodiments described herein.

FIG. 12 is a flowchart illustrating a process for generating regularexpressions and extracting data based on a capture group, via user dataselections received within a user interface, according to one or moreembodiments described herein.

FIG. 13 is an example user interface screen showing a tabular datadisplay, according to one or more embodiments described herein.

FIGS. 14 and 15 are example user interface screens illustrating thegeneration of regular expressions and capture groups based on selectionof data from a tabular display, according to one or more embodimentsdescribed herein.

FIGS. 16A and 16B are example user interface screens illustrating thegeneration of regular expressions based on selection of positive andnegative examples from a tabular display, according to one or moreembodiments described herein.

FIG. 17 is another example user interface screen illustrating thegeneration of a regular expression and capture group based on selectionof data from a tabular display, according to one or more embodimentsdescribed herein.

FIG. 18 is a flowchart illustrating a process for generating regularexpressions, including optional spans, using a longest commonsubsequence (LCS) algorithm, according to one or more embodimentsdescribed herein.

FIG. 19 is an example diagram for generating regular expressions,including optional spans, using a longest common subsequence (LCS)algorithm, according to one or more embodiments described herein.

FIG. 20 is a flowchart illustrating a process for generating regularexpressions based on combinatoric executions of a longest commonsubsequence (LCS) algorithm, according to one or more embodimentsdescribed herein.

FIG. 21 is a block diagram illustrating components of an exemplarydistributed system in which various embodiments of the present inventionmay be implemented.

FIG. 22 is a block diagram illustrating components of a systemenvironment by which services provided by embodiments of the presentinvention may be offered as cloud services.

FIG. 23 is a block diagram illustrating an exemplary computer system inwhich embodiments of the present invention may be implemented.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of various embodiments of the present invention. It willbe apparent, however, to one skilled in the art that embodiments of thepresent invention may be practiced without some of these specificdetails. In other instances, well-known structures and devices are shownin block diagram form.

The ensuing description provides exemplary embodiments only, and is notintended to limit the scope, applicability, or configuration of thedisclosure. Rather, the ensuing description of the exemplary embodimentswill provide those skilled in the art with an enabling description forimplementing an exemplary embodiment. It should be understood thatvarious changes may be made in the function and arrangement of elementswithout departing from the spirit and scope of the invention as setforth in the appended claims.

Specific details are given in the following description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, circuits,systems, networks, processes, and other components may be shown ascomponents in block diagram form in order not to obscure the embodimentsin unnecessary detail. In other instances, well-known circuits,processes, algorithms, structures, and techniques may be shown withoutunnecessary detail in order to avoid obscuring the embodiments.

Also, it is noted that individual embodiments may be described as aprocess which is depicted as a flowchart, a flow diagram, a data flowdiagram, a structure diagram, or a block diagram. Although a flowchartmay describe the operations as a sequential process, many of theoperations can be performed in parallel or concurrently. In addition,the order of the operations may be re-arranged. A process is terminatedwhen its operations are completed, but could have additional steps notincluded in a figure. A process may correspond to a method, a function,a procedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

The term “computer-readable medium” includes, but is not limited tonon-transitory media such as portable or fixed storage devices, opticalstorage devices, and various other mediums capable of storing,containing or carrying instruction(s) and/or data. A code segment orcomputer-executable instructions may represent a procedure, a function,a subprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment may be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents. Information, arguments,parameters, data, etc. may be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc.

Furthermore, embodiments may be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments to perform the necessarytasks may be stored in a machine readable medium. A processor(s) mayperform the necessary tasks.

Various techniques (e.g., methods, systems, non-transitorycomputer-readable storage memory storing a plurality of instructionsexecutable by one or more processors, etc.) are described herein forgenerating regular expressions corresponding to patterns identifiedwithin one or more input data examples. In certain embodiments, inresponse to receiving selections of input data, one or more patterns inthe input data are automatically identified and a regular expression (or“regex” for short) may be automatically and efficiently generated torepresent the identified patterns. Such patterns may be based uponsequences of characters (e.g., sequences of letters, numbers, spaces,punctuation marks, symbols, etc.). Various embodiments are describedherein, including methods, systems, non-transitory computer-readablestorage media storing programs, code, or instructions executable by oneor more processors, and the like.

In some embodiments, regular expressions may be composed using asymbolic wildcard-matching language, in order to match character stringsand/or extract information from character strings provided as input. Forinstance, a first example regular expression [A-Za-z] {3} \d?\d,\d\d\d\d may match certain dates (e.g., Apr. 3, 2018), and a secondexample regular expression [A-Za-z] {3} \d?\d, (\d\d\d\d) may be used toextract the year from matching dates. Input data received by a regularexpression generator system may include, for example, one more“positive” data examples, and/or one or more “negative” data examples.As used herein, a positive example may refer to a character sequencereceived as input that is to be matched by a regular expressiongenerated based on the input. In contrast, a negative example may referto an input character sequence that is not to be matched by a regularexpression generated based on the input.

A number of technical advantages may be realized within the variousembodiments and examples described herein. For example, certaintechniques described in this disclosure may improve speed and efficiencyof regular expression generation processes (e.g., regex solutions may begenerated in less than a second, and user interfaces may be suitable forinteractive real-time use). Various techniques described herein also maybe deterministic, may require no training data, may produce a solutionwithout requiring any initial regular expression input, and may becompletely automated (e.g., generating regular expressions withinrequiring any human intervention). Furthermore, various techniquesdescribed herein need not be limited regarding the types of data inputsthat may be handled effectively, and such techniques may improve humanreadability of the resulting regular expressions.

Certain embodiments described herein include one or more executions of aLongest Common Subsequence (LCS) algorithm. LCS algorithms may be usedin some contexts as difference engines (e.g., the engine behind the Unix“diff” utility) which are configured to determine and show differencesbetween two text files. In some embodiments, input data (e.g., stringsand other character sequences) may be converted into abstract tokens,which then may be provided as inputs to an LCS algorithm. Such abstracttokens may be for example, tokens based upon regular expression codes(e.g., Loogle codes or other character class codes) representing regularexpression character classes. Various different examples of such codesare possible, and may be referred to herein as “regular expressioncodes” or “intermediate regular expression codes” (IRECs). For example,an input character sequence “May 3” may be converted to the IREC code“LLLZN,” after which the tokenized string may be provided with othertokenized strings to the LCS algorithm. In some embodiments, an IREC(e.g., regular expression codes) that the input character sequences donot have in common, may appear in the final generated regular expressionas optional (e.g., an optional span). In certain embodiments, regularexpression codes may be category codes based upon the Unicode categorycodes shown athttps://www.regular-expressions.info/unicode.html#category. Forinstance, the code L may represent letters, the code N may representnumbers, the code Z may represent spaces, the code S may representsymbols, the code P may represent punctuation, and so on. For example,the code L may correspond to Unicode \p{L} and the code N may correspondto Unicode \p{N}. This allows for working one-to-one mappings from theLCS output to regular expressions (e.g. \pN\pN\pZ\pL\pL can match “10am”), which may provide advantages for human readability. Additionally,these different categories may be disjoint, or mutually exclusive. Thatis, in this example, the categories L, N, Z, P, and S may be disjointedso that there may be no overlap between members of the categories.

Additional technical advantages may be realized in various embodiments,including more efficient generation of regular expressions based on theuse of regular expressions codes (e.g., category codes), spans, etc. Byusing such codes, computing resources need not be wasted when the LCSalgorithm successfully identifies all or substantially all of thecharacters in the input strings as being different. Further technicaladvantages provided by the various embodiments herein include improvedreadability of the generated regular expressions, as well as supportingboth positive and negative examples as input data, and providing variousadvantageous user interface features (e.g., allowing the user tohighlight text fragments within a larger character sequence or data cellfor extraction).

I. General Overview

Various embodiments disclosed herein are related to generation ofregular expressions. In some embodiments, a data processing systemconfigured as a regular expression generator may generate a regularexpression, by identifying a longest common subsequence (LCS) that isshared by different sets of regular expression codes (e.g., categorycodes). Each set of regular expression codes may be converted fromsequence of characters received as input data through a user interface.Among the technical advantages described herein, abstracting input dataas intermediate codes (e.g., regular expression codes, spans, etc.) mayenable efficient generation of regular expressions using very littleinput data.

FIG. 1 is a block diagram illustrating components of an exemplarydistributed system for generating regular expressions, in which variousembodiments may be implemented. As shown in this example, a clientdevice 120 may communicate with a regular expression generator server110 (or regular expression generator) and interact with a user interfaceto retrieve and display tabular data, and generate regular expressionsbased on the selection of input data (e.g., examples) via the userinterface. In some embodiments, a client device 120 may communicate witha regular expression generator 110 via a client web browser 121 and/or aclient-side regular expression application 122 (e.g., client-sideapplication that receives/consumes regular expressions generated by aserver 110). Within the regular expression generator 110, requests fromclient devices 120 may be received over various communication networksat an network interface and processed by an application programminginterface (API), such as a REST API 112. A user interface data modelgenerator 114 component with the regular expression generator 110 mayprovide the server-side programming components and logic to generate andrender the various user interface features described herein. Suchfeature may include the functionality to allow users to retrieve anddisplay tabular data from data repositories 130, select input dataexamples to initiate the generation of regular expressions, and modifyand/or extract data based the regular expressions generated. In thisexample, a regular expression generator component 116 may be implementedto generate regular expressions, including converting input charactersequences into regular expression codes and/or spans, executingalgorithms (e.g., LCS algorithms) on input data, andgenerating/simplifying regular expressions. The regular expressionsgenerated by the regular expression generator 116, may be transmitted bythe REST service 112 to the client device 120, where Javascript code onthe client browser 121 (or corresponding client-side applicationcomponents 122) may then apply the regular expression against every cellin the spreadsheet column rendered in the browser. In other cases, aseparate regular expression engine component may be implemented on theserver-side to compare the generated regular expressions with thetabular data displayed on the user interface and/or within other datastored in data repositories 130, in order to identify matchingdata/non-matching data on the server-side. In various embodiments, thematching/non-matching data may be automatically selected (e.g.,highlighted) within the user interface, and may be selected forextraction, modification, deletion, etc. Any data extracted or modifiedvia the user interface, based on the generation of the regularexpressions, may be stored in one or more data repositories 130.Additionally, in some embodiments, the regular expressions generated(and/or corresponding inputs to the LCS algorithm) may be stored in aregular expression library 135 for future retrieval and use. In someembodiments, the generated regular expressions need not actually bestored in a “library,” buy may be incorporated into a “transformscript”. For examples, as described in more detail in U.S. Pat. No.10,210,246 (which is incorporated herein by reference for all purposes),such transform scripts may include programs, code, or instructions thatmay be executable by one or more processing units to transform receiveddata. Other possible examples of transform script actions may include“rename column”, “uppercase column data”, or “infer gender from firstname and create a new column with gender”, etc.

FIG. 2 is a flowchart illustrating a process for generating regularexpressions based on input received via a user interface, according toone or more embodiments described herein. In step 201, the regularexpression generator 110 may receive a request from a client device 120to access a regular expression generator user interface, and to viewparticular data via the user interface. The request in step 201 may bereceived via the REST API 112, and/or a web server, authenticationserver, or the like, and the user's request may be parsed andauthenticated. For instance, a user within an business or organizationmay access the regular expression generator 110 to analyze and/or modifytransaction data, customer data, performance data, forecast data, and/orany other categories of data that may be stored in the data repositories130 of the organization. In step 202, the regular expression generator110 may retrieve and display the requested data via a user interfacethat supports generation of regular expressions based on selected inputdata. Various embodiments and examples of such user interfaces aredescribed in detail below.

In step 203, a user may select one or more input character sequences,from the data displayed in the user interface provided by the regularexpression generator 110. In some embodiments, the data may be displayedin tabular form within the user interface, including labeled columnswith specific data types and/or categories of data. In such cases, theselection of input data in step 203 may correspond to a user selecting adata cell, or selecting (e.g., highlighting) an individual text fragmentwithin a data cell. However, in other embodiments, the regularexpression generator 110 may support retrieval and display ofsemi-structured and unstructured data via the user interface, and usersmay select input data for regular expression generation by selectingcharacter sequences from the semi-structured or unstructured data. Asdescribed below in examples, the user selecting input charactersequences from the tabular data displayed is just one example use case.In other examples, a user (e.g., a software developer or power userperhaps trying to compose a regular expression for the Linux commandline tools grep, sed, or awk, etc.) may type in examples from scratchrather than picking them off a spreadsheet.

In step 204, the regular expression generator 110 may generate one ormore regular expressions based on the input data selected by the user instep 203. In step 205, the regular expression generator 110 may updatethe user interface, for example, to display the generated regularexpression and/or to highlight matching/non-matching data within thedisplayed data. In step 206, which may be optional in some embodiments,the user interface may support functionality to allow the user to modifythe underlying data based on the generated regular expression. Forexample, the user interface may support features to allow the user tofilter, modify, delete, or extract particular data fields from thetabular data, based on whether those fields match or do not match theregular expression. Filtering or modifying data may include modifyingthe underlying data stored in the repositories 130, and in some cases,extracted data may be stored in a repository 130 as new columns and/ornew tables.

Although these steps illustrate a general and high-level overview of anexample user interaction with the user interface of the regularexpression generator 110, various additional features andfunctionalities may be supported in other embodiments. For example, insome embodiments, a regular expression code (or category code) may beassociated with a minimum number of occurrences of the code.Additionally or alternatively, the regular expression code may beassociated with a maximum number of occurrences of the code. As anexample, a set of regular expression codes may include the code L<0,1>to indicate that a particular portion of an LCS includes a letter eitherat least zero times, and at most once.

Additionally, in some embodiments, the input data may include three ormore character sequences. In such embodiments, techniques may be used todetermine order for performing the LCS algorithms on the three or morecharacter sequences, so that the resulting regular expression may begenerated in a performant manner to avoid the exponential increase inruntime caused by the three or more input character sequences. Theregular expression generator 110 may instead perform an LCS algorithm ontwo character sequences at a time, and may determine an order forselecting the pair of character sequences based on a graph. For example,a fully-connected graph may indicate that a first execution of the LCSalgorithm (e.g., LCS1) should be performed for Sequence1 and Sequence3,and then a second execution of the LCS algorithm (e.g., LCS2) should beperform for LCS1 and Sequence2, and so on. The graph may be afully-connected graph, with nodes representing the character sequences,and edges connecting the nodes to represent the length of an LCS sharedby the connected nodes. Each node in the graph may be connected to everyother node in the graph, and the order for selecting the charactersequences may be determined by a performing a depth-first traversal of aminimum spanning tree for the graph.

In further embodiments, input data may be provided via the userinterface in a number of different ways. For example, the input data mayindicate a first user selection of one or more characters within asecond user selection of a set of characters. For instance, a user mayhighlight a character within a set of previously highlighted characters.Thus, a second user selection may provide context for the first userselection, which may enable input data to be provided to the regularexpression generator 110 with greater specificity. In some embodiments,the regular expression generator 110 may generate and display, innear-real-time, a regular expression in response to each user selection.For example, when a user highlights a first range of characters, theregular expression generator 110 may display a regular expressionrepresenting the first range of characters. Then, when the userhighlights a second range of characters within the first range ofcharacters, the regular expression generator 110 may update the regularexpression that is displayed.

Additionally, in some embodiments, the regular expression generator 110may generate regular expressions based on input comprising both positiveand negative examples. As noted above, a positive example may refer to asequence of characters that are to be encompassed by a regularexpression, and a negative example may refer to a sequence of charactersthat are not to be encompassed by the regular expression. In such cases,the regular expression generator 110 may identify a shortest subsequenceof one or more characters, at a particular location, that distinguishthe positive example(s) from the negative example(s). The shortestsubsequence then may be hard-coded within the regular expression that isgenerated by the regular expression generator 110. In various examples,the shortest subsequence may be included in a prefix/suffix portion, ormid-span within the negative example(s).

Further examples for automatically generating regular expressionsaccording to certain embodiments are described below. These examples maycorrespond to various specific possible implementations of the generaltechnique in FIG. 2 , and be implemented in software (e.g., code,instructions, programs, etc.) executed by one or more processing units(e.g., processors, cores) of the respective systems, hardware, orcombinations thereof. The software may be stored on a non-transitorystorage medium (e.g., on a memory device). The further examplesdescribed below are intended to be illustrative and non-limiting.Although these examples depict the various processing steps occurring ina particular sequence or order, this is not intended to be limiting. Incertain alternative embodiments, the steps may be performed in somedifferent order or some steps may also be performed in parallel.

In some examples, the user inputs received via the user interface (e.g.,step 203) may include one or more “positive examples” to be matched bythe regular expression output, and zero or more “negative examples” thatare not to be matched by the regular expression output. Optionally, oneor more of the positive examples may be highlighted to select aparticular range (or subsequence) of characters. In some cases, in step204, the positive examples received via the user interface may beconverted to spans of regular expression codes (e.g., character categorycodes such as Unicode category codes). For each positive example, asequence of spans may be generated. A graph may be created in someembodiments, where each vertex corresponds to one of the sequences ofspans, and the edge weight equals the length of the output from the LCSalgorithm executed on those two sequences of spans corresponding to theendpoints of the edge. A minimum spanning tree may be determined for thegraph. For example, Prim's algorithm may be used in some embodiments toobtain a minimum spanning tree. A depth-first traversal may be performedon the minimum spanning tree to determine a traversal order, after whichthe LCS algorithm may be executed on the first two elements of thetraversal. Then, one by one, each additional element of the traversalmay be merged in order into the current LCS output, by executing the LCSalgorithm again on the output of the previous LCS iteration and the nextcurrent traversal element. The final output of the LCS algorithm, whichmay be a sequence of spans, then may be converted into a regularexpression. The conversion may be a one-to-one conversion in someembodiments, while certain optional embellishments described hereinmight not correspond to one-to-one conversions. Finally, the resultingregular expression may be tested against all positive and negativeexamples received via the user interface in step 203. If any of thetests fail, then the aforementioned process may be repeated using allthe positive examples and any negative examples that failed.

II. Regular Expression Generation Using Longest Common SubsequenceAlgorithm on Regular Expression Codes

As noted above, certain aspects described herein relate to generation ofregular expressions based upon the calculation of a longest commonsubsequence (LCS) shared by different sets of regular expression codescorresponding to input data.

FIG. 3 is a flowchart illustrating a process for generating regularexpressions using an LCS algorithm on sets of regular expression codes,according to one or more embodiments described herein. In step 301, theregular expression generator 110 may receive one or more charactersequences as input data. As noted above, in some examples, the inputdata may correspond to positive example data selected from within thetabular data displayed in the user interface, although it should beunderstood that the user interface is optional in some embodiments, andthe input data may correspond to any character sequence received anyother communication channel (e.g., non-user interface) in variousexamples.

In step 302, each character sequence received in step 301 may beconverted into a corresponding regular expression code. In variousembodiments, the regular expression codes may be Loogle codes, Unicodecategory codes, or any other character class codes representing regularexpression character classes. For example, an input character sequence“May 3” may be converted to the Loogle code “LLLZN.” In someembodiments, regular expression codes may be category codes based uponthe Unicode category codes shown athttps://www.regular-expressions.info/unicode.html#category. Forinstance, the code L may represent letters, the code N may representnumbers, the code Z may represent spaces, the code S may representsymbols, the code P may represent punctuation, and so on. For example,the code L may correspond to Unicode \p{L} and the code N may correspondto Unicode \p{N}.

In step 303, a longest common subsequence may be determined from amongthe sets of regular expression codes generated in step 302. In someembodiments, an LCS algorithm may be executed using two sets of regularexpression codes as input. Various different characteristics of theexecution of the LCS algorithm (e.g., direction of processing,anchoring, pushing spaces, coalescing low cardinality spans, aligning oncommon tokens, etc.), may be used in different embodiments. In step 304,a regular expression may be generated based on the output of the LCSalgorithm. In some cases, step 304 may include capturing the output ofthe LCS algorithm in regular expression codes, and converting theregular expression codes into a regular expression. In step 305, theregular expression may be simplified and output, for example, bydisplaying the regular expression for the user via the user interface.

FIG. 4 is an example diagram for generating a regular expression basedon two character sequence examples, using a longest common subsequence(LCS) algorithm on sets of regular expression codes. Thus, FIG. 4 showsan example of applying the process discussed above in FIG. 3 . As shownin FIG. 4 , the regular expression in this example is generated based onthe two input strings: “iPhone 5” and “iPhone X.” Each sequence in thisexample may be converted into a respective set of regular expressioncodes. Thus, iPhone 5 may be converted into “LLLLLLZN,” and iPhone X maybe converted into “LLLLLLZL.” As shown in FIG. 4 , these category codesare then provided as input to an LCS algorithm, which determines thatboth sets of IRECs (or category codes) comprise six Ls and one Z.Category codes excluded from the LCS may be represented as optionaland/or alternatives. Thus, a regular expression that encompasses bothcharacter sequences may be represented as the following: \pL{6}\pZ\pN?\pL? In this example, the regular expression includes Unicodecategory codes (e.g., \pL for letters, \pZ for spaces, and \pN fornumbers). The curly braces containing the number 6 indicates sixinstances of a letter, and the question marks indicate that anumber/letter at the end are optional. Finally, a simplification processmay be executed by the regular expression generator, during which theregular expression is simplified by inserting the common text fragment“iPhone” back into the final regular expression, replacing the broader“\pL{6}\” portion of the regular expression.

As shown in this example, the input strings received by the regularexpression generator 110 may be converted into “regular expressioncodes” representing regular expression broad categories (which also maybe referred to as “category codes”), and the LCS algorithm may be run onthose regular expression codes. In some embodiments, the Unicodecategory codes may be used for the regular expression codes. Forexample, an input text string may be converted into codes representingregex Unicode broad categories (e.g., \pL for letters, \pP forpunctuation, etc.). This approach, illustrated by FIGS. 3 and 4 may bereferred to as the indirect approach. However, in other embodiments, adirect approach may be used, in which the LCS algorithm is run directlyon the character sequences received as input.

In some embodiments, the indirect approach may provide additionaltechnical advantages, in that it need not require large amounts oftraining data, and may generate an effective regular expression with arelatively lower number of input examples. This is because the indirectapproach employs heuristics to reduce the uncertainty in the regularexpression generation, and to eliminate potential false positives andfalse negatives. For example, in generating a regular expression basedon the input strings “May 3” and “April 11,” the direct approach mayneed at least one example for every month to generate an effectiveregular expression matching date patterns. Relying on only those twoexamples, the direct approach may generate a regex of “[Am] [op] [yr][13] 1?” In contrast, the indirect approach, based on Unicode broadcategories, may generate a more effective regular expression of “\pL{3}\ d{1, 2}”. Additionally, as noted above, one of the technicaladvantages described herein includes efficient generation of regularexpressions using very little input data, even potentially from a singleexample. For instance, regarding generation of a regular expression fromthe single example “am”, a heuristic may determine whether to generate“am” or “\pL\pL” for the regular expression. Either is arguably correct,but so a programmed heuristic may implement user preferences and/orcriteria to determine how to generate an optimal regular expression(e.g., whether or not it should match “pm” as well).

Additionally, the indirect approach may further simplify the generatedregular expression “\ {3} \d{1,2}” to “[A-Za-z] {3} \d{1,2}” to make itmore human-readable. This may be beneficial in some embodiments, such aswhen outputting to non-sophisticated regular expression users who mightnot be familiar with the Unicode expressions for regular expressions.

Further, in some embodiments, instead of treating each characterindependently when executing the LCS algorithm, sequential and equalregular expression codes may be converted into span data structures(which also may be referred to as spans). In some cases, a span mayinclude a representation of single regular expression code (e.g.,Unicode broad category code), along with a repetition count range (e.g.,a minimum number and/or a maximum number). Conversion from regularexpression codes into spans may facilitates some various additionalfeatures described below, such as recognizing alternations (e.g.,disjunctions), and also may facilitate merging of adjacent optionalspans to further simplify the generated regular expressions.

As noted above, the LCS algorithm may be configured to store and retainthe underlying text fragments within the input character sequences,which may potentially be inserted back into the final regularexpression, such as the string “iPhone” in FIG. 4 . By keeping track ofthe text fragments that originally gave rise to the category codeassigned to that span, such embodiments may allow for literal text(e.g., am and pm) to be included directly in the generated regularexpression, which may reduce false positives and make the regularexpression output more human readable.

III. Regular Expression Generation Using Longest Common SubsequenceAlgorithm on Combinations of Regular Expression Codes

Additional aspects described herein relate to the generation of regularexpressions based on input data comprising three or more strings (e.g.,three or more separate character sequences). When three or more stringsare identified as input data, the regular expression generator 110 mayuse a performance optimization feature in which an optimal order isdetermined for the sequence of LCS algorithm executions. As discussedbelow, the performance optimization feature for more than two stringsmay involve building a graph with a vertex corresponding to each string,and edge lengths/weights which may be based on the size of the LCSoutput between each string and every other string. A minimum spanningtree then may be derived using those edge weights, and a depth-firsttraversal may be performed to determine an order of the input strings.Finally, the series of LCS algorithms may be done using the determinedorder of input strings.

FIG. 5 is a flowchart illustrating a process for generating regularexpressions using a longest common subsequence (LCS) algorithm on largersets (e.g., three or more character sequences) of regular expressioncodes. Thus, steps 502-505 in this example may correspond to step 303discussed above in FIG. 3 . However, because this example relates togenerating regular expressions based on three or more input charactersequences, the LCS algorithm may be performed multiple times. Forexample, in order to avoid an exponential increase in runtime for threeor more input strings, the LCS algorithm may be executed multiple times,wherein each execution is performed on only two input strings. Forexample, the regular expression generator 110 may perform an initialexecution of the LCS algorithm on two strings (e.g., two input charactersequences or two converted regular expression codes), then may perform asecond execution of the LCS algorithm on the output of the first LCSalgorithm and a third string, and then may perform a third execution ofthe LCS algorithm on the output of the second LCS algorithm and a fourthstring, and so on.

In order to improve and/or optimize the performance of such embodiments,it may be desirable to determine an optimal order for the input strings(e.g., input character sequences or regular expression codes) to performthe sequence of LCS algorithms. For example, a good order for taking theinput strings may affect the readability of the generated regularexpression, such as by minimizing the number of optional spans. To keepthe generated regex concise, additional strings that are LCS'd into thecurrent regex should preferably already be somewhat similar to thecurrent regex (the intermediate result from LCS'ing the already-seenstrings).

Thus, in step 501, the plurality (e.g., 3 or more) input charactersequences are converted into regular expression codes. In step 502, anorder is determined for processing the regular expression codes usingthe LCS algorithm. The determination of the order in step 502 isdiscussed more below in reference to FIG. 7 . In step 503, either thefirst two regular expression codes in the determined order are selected(for the first iteration of step 503), or the next regular expressioncodes in the determined order is selected (for subsequent iterations ofstep 503). In step 504, the LCS algorithm is executed on two inputstrings corresponding to the format of regular expression codes. For thefirst iteration of step 504, the LCS algorithm is executed on the firsttwo regular expression codes in the determined order, and for subsequentiterations of step 504, the LCS algorithm is executed on the nextregular expression code in the determined order and the output of theprevious LCS algorithm (which also may be in same format of regularexpression codes). In step 505, the regular expression generator 110determines whether or not there are additional regular expression codesin the determined order that have not yet been provided as input to theLCS algorithm. If so, the process returns to step 503 for anotherexecution of the LCS algorithm. If not, in step 506, a regularexpression is generated based on the output of the last execution of theLCS algorithm.

FIG. 6 is an example diagram for generating a regular expression basedon five input character sequence examples. In this example, each inputcharacter sequence is converted to a regular expression code, and thenan LCS algorithm is executed repeatedly based on a determined order ofthe regular expression codes. Thus, FIG. 6 shows one example of applyingthe process discussed above in FIG. 5 . In this example, the determinedorder for the five regular expression codes is Code #1 to Code #5, andeach codes is input to the LCS algorithm in the determined order togenerate a regular expression output. The final regular expressionoutput (Reg Ex #4) corresponds to the final regular expression generatedbased on all five of the input character sequences.

FIG. 7 is a flowchart illustrating a process for determining an order ofexecution for a longest common subsequence (LCS) algorithm on largersets (e.g., three or more) of regular expression codes. Thus, as shownin this example, steps 701-704 may correspond to the order determinationin step 502, discussed above. In step 701, the LCS algorithm may be runon each unique pair of regular expression codes corresponding to theinput data, and the resulting output LCS may be stored for eachexecution. Thus, for k number of input data, this may represent all(k(k−1))/2 possible pairings of strings to be run through the LCSalgorithm, or k(k−1) in some embodiments. For example, if k=3 inputcharacter sequences are received, LCS algorithm may be run three timesin step 701; if k=4 input character sequences are received, the LCSalgorithm may be run six times in step 701; if k=5 input charactersequences are received, the LCS algorithm may be run ten times in step701, and so on. In step 702, a fully-connected graph may be constructedof k nodes representing the strings with the edge weight of the(k(k−1))/2 edges being the length of the raw LCS output between the twonodes. In step 703, a minimum spanning tree may be derived from thefully-connected graph in step 702. In step 704, a depth-first traversalmay be performed on the minimum spanning tree. The output of thistraversal may correspond to the order in which regular expression codeswill be input into the sequence of LCS algorithm executions.

Referring briefly to FIGS. 8A and 8B, an example of a fully-connectedgraph is shown in FIG. 5 , generated based on k=5 input charactersequences received, and in FIG. 8B a minimum spanning treerepresentation is shown for the fully-connected graph.

In some embodiments, the approach described in FIGS. 5-8B may provideadditional technical advantages with respect to performance. Forexample, certain conventional implementations of the LCS algorithm mayexhibit a run-time performance of O(n²) where n is the length of thestrings. Extending such implementations to k strings instead of only 2,may results in an exponential run-time performance O(n^(k)), because theLCS algorithm may be required to search a k-dimensional space. Suchconventional implementations of the LCS algorithm might not beperformant or sufficiently suitable for real-time on-line userexperiences.

As noted above, the LCS algorithm may be executed (k(k−1))/2 times,where sometimes the duplicates are the very same as have been seenbefore, because the LCS algorithm may when the raw input examples fromthe user have been converted to regex category codes. Thus, memorizationmay be implemented in some cases, in which a cache can be used to mappreviously-seen LCS problems to the previously worked LCS solution.

IV. Regular Expression Generation Based on Positive and Negative PatternMatching Examples

Additional aspects described herein relate to generating regularexpressions based on input data corresponding to both positive andnegative examples. As noted above, a positive example may refer to aninput data character sequence that is designated as an example stringthat should match the regular expression that will be generated by theregular expression generator. In contrast, a negative example may referto an input data character sequence that is designated as an examplestring that should not match the regular expression that will begenerated by the regular expression generator. As discussed below, insome embodiments, the regular expression generator 110 may be configuredto identify a location and a shortest subsequence of characters at thelocation that distinguish the positive examples from the negativeexamples. The shortest subsequence then may be hard-coded into thegenerated regular expression, so that the positive examples will matchthe regular expression and the negative examples will be excluded by(e.g., will not match) the regular expression.

FIG. 9 is a flowchart illustrating a process for generating a regularexpression based on positive and negative character sequence examples.In step 901, the regular expression generator 110 may receive one ormore input data character sequences corresponding to positive examples.In step 902, the regular expression generator 110 may generate a regularexpression based on the received positive examples. Thus, steps 901-902may include some or all of the steps performed in FIG. 3 or FIG. 5 ,discussed above, to generate a regular expression based on input datacharacter sequences.

In step 903, the regular expression generator 110 may receive oneadditional input data character sequences corresponding to negativeexamples. Thus, the negative examples are specifically designated so asnot the match the regular expression generated in step 902. In someembodiments, the negative examples received in step 903 may be initiallytested against the regular expression generated in step 902, and if itis determined that the negative examples do not match the regularexpression, then no further action is taken. However, in this example itmay be assumed that at least one of the negative examples received instep 903 matches the regular expression generated in step 902. Thus, instep 904, a disambiguation location may be determined within the regularexpression generated in step 902. In some embodiments, thedisambiguation location may be selected as either the prefix location(e.g., at the beginning of the regular expression) or the suffixlocation (e.g., at the end of the regular expression). For instance, theregular expression generator 110 may determine a first number ofcharacters that would be needed at the prefix to distinguish thepositive examples from the negative examples, and second number ofcharacters that would be needed at the suffix to distinguish thepositive examples from the negative examples. The regular expressiongenerator 110 may then select the suffix or prefix based on the shortestnumber of replacement characters needed. In some cases, using the prefixas the disambiguation location may be preferred (e.g., weighted) forreadability purposes. In still other examples, the disambiguationlocation may be a mid-span location that does not correspond to theprefix or suffix of the regular expression.

In step 905, the regular expression generator 110 may determine areplacement sequence of custom character classes which, when insertedinto the regular expression at the determined location, may distinguishthe positive examples from the negative examples. In some embodiments,the regular expression generator 110 in step 905 may retrieve textfragments from each of the positive and negative examples, correspondingto the disambiguation location (or replacement location), and then usethe text fragments to determine a discriminator to be used as areplacement sequence that distinguishes the positive examples from thenegative examples. Additionally, the discriminator replacement sequencedetermined in step 905 may include multiple different replacementsequences of custom character classes, which may be replaced either atthe same location or at different locations within the regularexpression.

As noted above, in some cases, the determination of the replacementsequence in step 905 may be performed in conjunction with thedetermination of the disambiguation location (or replacement location)in step 904. For example, the regular expression generator 110 maydetermine one or more replacement sequences which, at a first possiblereplacement location, may distinguish the positive from the negativeexamples. The regular expression generator 110 also may determine one ormore other replacement sequences which, at a second different possiblereplacement location, may distinguish the positive from the negativeexamples. In this example, when selecting between the different possiblereplacement locations and corresponding replacement sequences, theregular expression generator 110 may apply a heuristic formula toperform the selection based on one or more of the sizes in characters ofthe replacement locations, and the numbers and/or sizes of thecorresponding replacement sequences. Finally, in step 906, the regularexpression may be modified by inserting the one or more determinedreplacement sequences into the determined location to replace theprevious portion of the regular expression. In some cases, following themodification of the regular expression in step 906, the positive and/ornegative examples may be tested against the modified regular expressionto confirm that the positive examples match and that the negativeexamples do not match the regular expression.

FIGS. 10A and 10B are example user interface screens showing generationof regular expressions based on positive and negative character sequenceexamples. Thus, the example shown in FIGS. 10A and 10B may correspond tothe user interfaces displayed during the execution of the process ofFIG. 9 discussed above. In FIG. 10A, the user provides three positiveexamples of data input character sequences 1001, and the regularexpression generator 110 generates a regular expression 1002 thatmatches each of the positive examples. Then, in FIG. 4B, the userprovides one negative example 1004, and the regular expression generator110 generates a modified regular expression 1005, which is based on boththe current sets of positive examples 1003 and negative examples 1004.

As noted above, in some embodiments, when both positive and negativeexamples are received, the regular expression generator 110 may identifya discriminator, or the shortest subsequence of one or more charactersthat distinguish the positive example(s) from the negative example(s).The selected discriminator may be a shortest sequence (e.g., expressedin category codes), and may either be positive or negative, so that thepositive examples will match and the negative examples will not. In somecases, the discriminator may correspond to a replacement subsequencewhich then may be hardcoded into the regular expression in step 905. Asan example, in “[AL][a-z]+” the [AL] is a positive discriminator that,assuming it is applied to street suffixes, would match with (or allow)“Alley”, “Avenue”, and “Lane” but would not match with (or disallow)everything else. As another example, in “[BC][o][a-z]+” the [BC][o] is apositive discriminator consisting of a sequence of two character classesthat would match with “Boulevard” and “Court”. As yet another example,in “[{circumflex over ( )}A][a-z]+” the [{circumflex over ( )}A] may bea negative discriminator that would disallow “Alley” and “Avenue”. Insome cases, the algorithm may make generate a negative-look-behind todiscriminate correctly. For example, (?<!Av)[A-Za-z]+ would exclude“Avenue” but would allow “Alley”.

As another example, if the user supplies the positive examples“202-456-7800” and “313-678-8900” and negative examples “404-765-9876”and “515-987-6570”, then in certain embodiments, the regular expressiongenerator 110 may generate the regular expression“\d\d\d-\d\d\d-\d\d00”. That is, the replacement character subsequencemay be identified for the suffix of the regular expression, based on thedetermination that phone numbers that end in 00 distinguish the positiveexamples from the negative examples (e.g., assuming that the goal is aregular expression the matches business phone numbers). This is anexample of negative example by suffix (or more specifically, an exampleof accomodating negative examples by using a positive suffix), butvarious other embodiments may support either replacements at prefixes,suffixes, or mid-span locations. In examples of replacement at mid-spanlocations, a character offset into the span may be kept track of, andmay be split at the mid-span point.

To decide between whether to use a prefix or suffix, in someembodiments, a heuristic is employed where the minimum score is chosenover all combinations of k_(a) and prefix/suffix:

${score} = {{k_{a}{\min^{2}\left\{ {\frac{F_{p}}{1 + {E_{p}}},\frac{F_{n}}{1 + {E_{n}}}} \right\}}} + \left\{ \begin{matrix}0.0 & {{if}\mspace{14mu}{prefix}} \\0.1 & {{if}\mspace{14mu}{suffix}}\end{matrix} \right.}$

-   -   Where:    -   k_(a)=number of characters being considered to disambiguate the        affix (prefix or suffix)    -   |F_(p)|=number of unique text fragments from the positive        examples required to disambiguate the affix    -   |F_(n)|=number of unique text fragments from the negative        examples required to disambiguate the affix    -   |E_(p)|=number of (complete) positive examples provided by the        user    -   |E_(n)|=number of (complete) negative examples provided by the        user

In the above example, the heuristic is designed to favor shorterdisambiguation text fragments over longer ones (e.g., thus themultiplication by k_(a)). The heuristic is also designed to favor theprefix over the suffix (e.g., thus the penalty of 0.1 for suffix), toimprove readability. Finally, the heuristic is designed to favordisambiguating (e.g., replacing) a longer prefix or suffix, overdisambiguating by using a larger number of string fragments (e.g., thusthe squaring of the number of string fragments to be replaced.

As noted above, some embodiments also may support negative mid-spanexamples as well as negative look-behind examples and negativelook-ahead examples.

Once a prefix/suffix and k (the number of characters to disambiguate)have been determined, the regular expression generator 110 still maydetermine how to represent that disambiguation in the generated regularexpression. The generated regular expression may be either permissivefor affixes (e.g., prefixes or suffixes) that look like the positiveexamples, or may exclude affixes that look like the negative examples.

${usePermissive} = {\frac{E_{p}}{F_{p}} - \frac{E_{n}}{F_{n}}}$If usePermissive is greater than zero, then things that look like thepositive examples are allowed through by generating regular expressionsthat allows characters, one by one for (each character position), takenfrom the positive examples. In other cases, the regular expressiongenerator 110 may take the approach of disallowing things that look likethe negative examples by generating a regular expression that disallowscharacters, one by one (for each character position), taken from thenegative examples.

As another example, a generated regular expression for the positiveexample 8 am and negative example 9 pm might be \d[{circumflex over( )}p]m. This uses the caret syntax. In some cases, the regularexpression generator 110 may be configured to favor a shorter regularexpression, which may be not only more readable to the user, but alsomay be more likely to be correct. The rationale is that a frequentlyappearing character is more likely to appear again in the future, and soan emphasis should be placed upon frequently appearing characters. Ifthere are fewer unique characters |F_(p)| (fewer unique because the onesthat do appear do so more frequently) then this is rewarded in theheuristic by having it in the denominator.

Referring again to the usePermissive example heuristic above,determining one unique positive affix is no big feat if there was onlyone positive example from the user. Thus, in this heuristic low |E_(p)|is penalized by having it in the numerator (i.e. high |E_(p)| isrewarded in this heuristic).

Additionally, in some embodiments, negative examples may be based onlook-behind and/or look-ahead. For example, the user may provide apositive example of “323-1234” and a negative example of “202-754-9876”then that involves use of the regex look-behind syntax (?<!) to excludephone numbers with area codes.

Negative examples also may be based on optional spans in some cases. Forexample, the user may provide positive examples of “ab” and “a2b” and anegative example of “a3b”. In this case, an example implementation mayfail, because it may attempt to discriminate based only on requiredspans and the “2” digit is in an optional span. In this example, failuremay refer to a situation in which the generated regular expressionmatches all of the positive examples (correctly) and also matches one ormore of the negative examples (erroneously). In such cases, the user mayalerted to the failure and may be provided the options, via the userinterface, to manually repair the generated regular expression and/or toremove some of the negative examples.

V. User Interface for Regular Expression Generation

Additional aspect described herein include several different featuresand functionality within a graphical user interface related togeneration of regular expressions. As discussed below, certain of thesefeatures may including various options for user selection andhighlighting for positive and negative examples, color-coding forpositive and negative examples, and multiple overlapping/nestedhighlighting within a data cell.

FIG. 11 is a flowchart illustrating a process for generating regularexpressions based on user data selections received within a userinterface. The example process in FIG. 11 may correspond to any of thepreviously discussed examples of generating regular expressions based oninput data character sequences. However, FIG. 11 describes the processwith respect to the user interface that may be generated and displayedon a client device 120. In step 1101, in response to a request from auser via the user interface, the regular expression generator 110 mayretrieve data (e.g., from a data repository 130) and render/display thedata in tabular form within a graphical user interface. Although tabulardata is used in this example, it should be understood that tabular dataneed not be used or displayed in other examples. For instance, in somecases a user may type in raw data directly (rather than selecting datafrom the user interface). Additionally, when data is presented on viathe user interface, the data need not be in tabular form, but may beunstructured data (e.g., a document) or semi-structured (e.g., aspreadsheet of unformatted/unstructured data items such as tweets orposts). In various examples, the tabular data may correspond transactiondata, customer data, performance data, forecast data, and/or any othercategories of data that may be stored in the data repositories 130 for abusiness or other organization. In step 1102, a user selection of inputdata may be received via the user interface. The selected input datamay, for example, correspond to an entire data cell selected by theuser, or a subsequence of characters within a data cell. In step 1103,the regular expression generator 110 may generate a regular expressionbased on the input data received in step 1102 (e.g., the data cell orportions thereof). In step 1104, the user interface may be updated inresponse to the generation of the regular expression. In some cases, theuser interface may be updated simply to display the generated regularexpression to the user, while in other cases the user interface may beupdated in various other ways as discussed below. As shown in thisexample, the user may select multiple different input data charactersequences via the user interface, and in response to each new input datareceived, the regular expression generator 110 may generate an updatedregular expression which encompasses both the first and second(positive) examples of character sequences. Then, when the userhighlights a third sequence of characters (e.g., outside of bothcharacter sequences, or within the first or second character sequence)the regular expression generator 110 may update the regular expressionagain, and so on. In some embodiments, the regular expression generator110 may execute the algorithm in real-time (or near real-time) so thatan entirely new regular expression may be generated in response to eachnew keystroke or each new highlighted section made by the user.

Thus, as shown in FIG. 11 , in response to user selections of charactersequences via the user interface, the regular expression generator 110may generate and display a regular expression. For example, when a userhighlights a first sequence of characters, the regular expressiongenerator may generate and display a regular expression representing thefirst sequence of characters. When the user highlights a second sequenceof characters, the regular expression generator may generate an updatedregular expression which encompasses both the first and second sequencesof characters. Then, when the user highlights a third sequence ofcharacters (e.g., within either the first or second sequence) theregular expression generator may update the regular expression again,and so on.

FIG. 12 is another flowchart illustrating a process for generatingregular expressions and extracting data based on a capture group, viauser data selections received within a user interface. In step 1201, asdiscussed above in step 1101, the regular expression generator 110 mayretrieve data (e.g., from a data repository 130) and render/display thedata in tabular form within the graphical user interface. In step 1202,the regular expression generator 110 may receive selection of userhighlighting of a text fragment within a particular data cell. In step1203, the regular expression generator 110 may generate a regularexpression based on the positive example of the selected data cell, andin step 1204 may create a regular expression capture group based on thetext fragment highlighted within the cell. In step 1205, the regularexpression generator 110 may determine one or more additional cellswithin the displayed tabular data that match the generated regularexpression, and in step 1206 the corresponding text fragments within theadditional cells that match the generated regular expression may beextracted.

Thus, in addition to supplying the positive examples, the user also mayselect (e.g., via mouse text highlighting) a text fragment within any ofthe selected positive examples. In response, the regular expressiongenerator 110 may create a regular expression capture group to extractthat text fragment from the example as well as the correspondingfragment from all other matches in the text the regular expression isapplied to. Extracting the text fragments from matching data cells alsomay include deleting and modifying, and may be used in some cases tocreate a new column of data out of an existing column of semi-structuredor unstructured text.

Using an example of a user selecting a positive data example, and if theuser highlighted the year, then the regular expression generator 110 maygenerate the regular expression (?:[A-Z]{3}\s+\d\d,\s+|\d\d/\d\d)(\d\d\d\d). As shown in this example, the regularexpression generator 110 has put parentheses around the year, and alsoconverted the old parentheses around the month and day (used foralternation) into a “non-capturing” group by use of the ?: regex syntax.In some embodiments, an extraction/capture group may be required to fallon span boundaries, and in such embodiments the regular expressiongenerator 110 may take the highlighted character range as input andexpands it to encompass the nearest anchor span boundaries. However, inother examples, the mid-span extraction/capture may be supported by theuser interface.

In some embodiments, the user interface may support input data from usesthat includes a selection of a first character sequence within a secondcharacter sequence. For instance, a user may highlight one or morecharacters within a larger previously highlighted character sequence,and the second user selection may provide context for the larger firstuser selection. Such embodiments may enable input data to be provided tothe regular expression generator 110 with greater specificity.

Additionally, in some examples, an operation may be initiated and adialog may be opened in response to a user selecting (e.g., highlightingtext) within the user interface. In some cases, the dialog may be anon-model dialog, such as floating toolbox window that does not preventuser interaction with the main screen. The dialog also may change inappearance and/or functionality depending on what major operation theuser is performing. Thus, in such cases, the user need not search for afurther menu item after highlighting the selected text, in order toinitiate the modification, extracting, etc., of the capture group textfragments. Additionally, in certain embodiments, the user interfaceprovided for generating regular expressions may include three highlightmodes: nested-auto, nested-manual, and single-level. In certain cases,the default mode of operation may be that the entire cell is identifiedas the highlighted region, and the user may further highlight one ormore additional subsequences within the highlighted cell. In othermodes, the user may be allowed to manually specify both highlightswithin a data cell of the tabular data display. In still other modes,the user may be allowed to manually specify an outer highlight with noinner highlight. These other modes may be better suited to“semi-structured” data, for example, a column of data consisting oftweets or other long strings such as browser “user agent” strings.“Semi-structured” data refers to data that may be displayed in tabularform within the user interface, but where a column within the tableconsists of unstructured text.

In some such embodiments, inner and outer selection (e.g., highlighting)by the user via the user interface may be distinguished by color coding.For example, the outer highlights of a positive example may be shown ina first text/background color combination, and the inner highlight of apositive example may be shown in a different contrasting text/backgroundcolor combination.

As indicated above, a user may specify a selection of a capture groupvia selection of a character subsequence. The GUI may be used tofacilitate user selection via highlighting (or other indications). Anexample is shown in FIG. 13 , in which an example user interface screenis shown with a tabular data display. In this example, FIG. 13 depictshighlighting within a column value, for example, caused by a userdragging a mouse across one or more desired elements of the columnvalue. Note that the “cell” in which the user highlighting is performedmay exhibit a color change indicating selection of the column value.This color change may be construed as automated highlighting responsiveto the user highlighting.

FIGS. 14 and 15 are example user interface screens illustrating thegeneration of regular expressions and capture groups based on selectionof data from a tabular display. In these examples, FIGS. 14 and 15 showan additional user interface window that be displayed automaticallydetection of user highlighting 1401 within the tabular data display. Thewindow comprises a field 1402 for displaying positive examples, a fieldfor displaying negative examples, and a field for displaying the regularexpression that is generated dynamically (and near-instantaneously) inresponse to the selection of positive examples form the tabular datadisplay. In these example, user highlighting within a column value 1401may be equivalent to user highlighting within automated highlighting.Thus, user highlighting of the area code causes not only theuser-highlighted area code 1401, but also the rest of the phone numberto be populated in the positive example field 1402.

However, it should be appreciated that user highlighting is not limitedto performance within automated highlighting. For example, userhighlighting may alternatively be performed within other userhighlighting. As another example, user highlighting may alternatively beperformed without any inner highlighting (e.g., further highlightingwithin highlighted text). These alternative examples are particularlysuitable for semi-structured data, such as a column of data comprising“Tweets” or other long strings (e.g., browser “user agent” strings).

Furthermore, upon generation of the corresponding regular expression,other column values 1402 matching the regular expression may beidentified based upon additional automated highlighting. In the examplesshown in FIGS. 14 and 15 , the additional automated highlightingindicates the elements of these other column values that match thecapture group of the generated regular expression. The additionalautomated highlighting may be performed using a color that is differentfrom the one used for the user highlighting.

As shown in FIG. 15 , additional user highlighting is shown to indicateuser selection of other examples. The additional user highlighting maybe performed in a manner similar to that described above. Thus, the userinterface in FIG. 15 shows the population of other examples in the field1502 for displaying positive examples. This may occur responsive todetection of the additional user highlighting. Additionally, thegenerated regular expression 1503 may be updated dynamically andnear-instantaneously, such that it matches all of the positive examples1502. Responsive to generation of the updated regular expression,automated highlighting of other column values 1504 matching the updatedregular expression may also be updated. In some implementations, dynamiccolor-coding also may be used. For instance, matches may be color-codedusing a first color (e.g., blue), while positive examples arecolor-coded using a second color (e.g., green), and negative examplesmay be color-coded using a third color (e.g., red).

FIGS. 16A and 16B are example user interface screens illustrating thegeneration of regular expressions based on selection of positive andnegative examples from the tabular display. In FIGS. 16A-16B, individualexamples from the positive examples field 1602 may be removed from thepositive examples field 1603, and/or moved to the negative examplesfield 1603. Within the user interface, this may be performed, forexample, by the user clicking (e.g., right-clicking) on one of theexamples to selecting it. The selection may cause the user interface todisplay a menu 1602 comprising a delete option and a change option.Thereafter, clicking on an option may cause performance of thecorresponding function.

In the example shown in FIGS. 16A and 16B, the result of the userselection of the change option, is to move the selected example is movedto the negative examples field 1603, causing the regular expression 1601to be updated to regular expression 1604, which may be generateddynamically and near-instantaneously (e.g., between 30 ms and 9000 ms incertain embodiments). Responsive to generation of the updated regularexpression 1604, the automated highlighting of other column valuesmatching the updated regular expression may also be updated within thetabular data display. Furthermore, automated highlighting may beperformed on some or all of the negative examples, including any columnvalues corresponding to the negative example, which may be highlightedusing a color that is different from any of the colors used above, orotherwise distinguished within the user interface using other visualtechniques.

In some embodiments, specifying a negative example via the userinterface need not require first specifying the example as a positiveexample, and then converting it into a negative example as shown inFIGS. 16A and 16B. Rather, a negative example may be specified in avariety of ways. For example, a user may select (e.g., right click) acolumn value via the user interface (e.g., one of the other columnvalues on which automated highlighting was performed to indicate that itmatches the generated regular expression), which may thereby causedisplay of a menu comprising an option (e.g., “Make New Counterexample”)to designate the selected column value as a negative example.

Thus, using the examples shown in FIGS. 16A and 16B, responsive togeneration of the updated regular expression 1604, automatedhighlighting of other column values matching the updated regularexpression may also be updated. In these examples, the updated regularexpression specifies telephone numbers that end “9”.

Returning briefly to FIGS. 14 and 15 , when the “Extract” button isclicked or otherwise selected by the user, an operation may be initiatedto extract the highlighted text fragments within all of the cellsmatching the current regular expression 1403 or 1503. Although not shownin FIGS. 14 and 15 , in some embodiments the user interface may provideother selectable buttons in addition to or instead of the “Extract”button. For example, a “Replace” button may be presented as an option toreplace user-highlighted elements with user-specified elements.Additionally or alternatively, one or more “Delete” buttons may bepresented as an option to, in effect, replace user-highlighted elementswith nothing. For instance, one or both of a “Delete Fragment” operationand/or a “Delete Row” operation may be implemented, which will deleteeither the user-highlighted text fragment or the either row,respectively. Additional operations that may be implemented in variousembodiments may include a “Keep Row” operation, a “Split” operation(e.g., highlight comma, then extract the comma-separated components intoseparate multiple new columns), and an “Obfuscate” operation (e.g.,replace highlighted text/capture group with a sequence of “#” or othersymbols).

In this example, in response to the selection of the “Extract” button,an extraction operation may be added to a list of transform scripts tobe performed by a downstream operation. In some embodiments, the list oftransform scripts may be displayed in a portion of the user interfacefor review/modification by the user. Alternatively, the extractionoperation may be performed in situ to generate a new column comprisingthe contents of the regex capture group (e.g., the elementscorresponding to the user-highlighted portions of a positive example).In the examples shown in FIGS. 14 and 15 , a new column and/or a newtable of area codes may be generated in response to a selection of the“Extract” button.

FIG. 17 is another example user interface screen illustrating thegeneration of a regular expression and capture group based on selectionof data from a tabular display, according to one or more embodimentsdescribed herein.

VI. Regular Expression Generation Using Longest Common SubsequenceAlgorithm on Spans

Additional aspects described herein relate to the generation of regularexpressions, based on the LCS algorithm from one or more data inputcharacter sequences, but wherein the regular expression generator 110also may handle characters that are present in only some of theexamples. To handle characters that are present in only some inputexamples, spans may be defined in which both a minimum and maximumnumber of occurrences of a regular expression code are tracked. Forexample, for the character sequence inputs of “9 pm” and “9 pm” anoptional space is present between the number and the “pm” text. In suchcases, when a certain span (e.g., the single space between “9” and “pm”)might not be present at all of the given input examples, the minimumnumber of occurrences may be set to zero. These minimum and maximumnumbers can then be mapped to the regular expression multiplicitysyntax. A longest common subsequence (LCS) algorithm may be run on thespans of characters derived from the input examples, including“optional” spans (e.g., minimum length of zero) which do not appear inevery input example. As discussed below, consecutive spans may be mergedduring the execution of the LCS algorithm. In such cases, when extraoptional spans that are being carried along end up appearingconsecutively, the LCS algorithm may be run recursively on thoseoptional spans as well. That is, although the running of the LCSalgorithm is by its nature recursive, in these cases the entire LCSalgorithm may be run recursively (e.g., recursively running therecursive LCS algorithm). Among other technical advantages, this mayallow for a shorter, cleaner, and more readable regular expressiongeneration. For instance, (am| am) (i.e., with optional space before theam) might be generated without recursively running the LCS algorithm,whereas recursively running the LCS algorithm may result in the regularexpression generated as (?am), which is shorter and cleaner.

FIG. 18 is a flowchart illustrating a process for generating regularexpressions, including optional spans, using a longest commonsubsequence (LCS) algorithm, according to one or more embodimentsdescribed herein. In step 1801, the regular expression generator 110 mayreceive one or more character sequences as input data, corresponding topositive regular expression examples. In step 1802, the regularexpression generator 110 may convert the character sequences intoregular expression codes. Thus, steps 1801 and 1802 may be similar oridentical to previous corresponding examples discussed above. Then, instep 1802, the regular expression codes may further by converted intospan data structures (or spans). As noted above, each span may include adata structure storing a character class code (e.g., a regex code) and arepetition count range (e.g., a minimum count and/or a maximum count).In step 1804, the regular expression generator 110 may execute an LCSalgorithm, providing the sets of spans as input to the algorithm. Theoutput of the LCS algorithm in this example may include an output set ofspans, including at least one span having a minimum repetition countrange equal to zero, which corresponds to an optional span within theoutput of the LCS algorithm. Finally, in step 1805, the regularexpression generator 110 may generate a regular expression based on theoutput of the output of the LCS algorithm, including the optional span.

FIG. 19 is an example diagram illustrating the generation of a regularexpression using a longest common subsequence (LCS) algorithm, whereinthe generated regular expression includes an optional span. In thisexample, the two input data character sequences are “8 am” and “9 pm”.The input data character sequences are first converted to regularexpression codes (step 1802) and then to spans (step 1803), as discussedabove. The spans may be provided as input to an LCS algorithm (step1804), and the LCS output includes the optional span Z indicating thatan optional single space may be number and the two-letter text sequence.That is, the superscript notation in this example may include the twonumbers, the minimum repetition count range (e.g., 0), and the maximumrepetition count range (e.g., 1) which apply to the preceding code(e.g., Z=spaces). Finally, the regular expression may be generated basedon the output span of the LCS algorithm, and the optional span may beconverted to the corresponding regular expression code “pZ*”.

In some embodiments, the rendition and use of optional space by theregular expression generator 110, during the execution of the LCSalgorithm, may provide additional technical advantages with respect toperformance and readability. For example, when generating regularexpressions, it is desirable in some cases to be able to handle both thecharacters that are in common amongst all the given examples, and thecharacters that are present in only some of the examples.

In certain embodiments, for each span data structure, both the minimumnumber of occurrences of a category code and a maximum number ofoccurrences of the category code may be tracked. In the case where aspan is not present at all in one or more of the given examples, theminimum is set to zero. As another example, to generate a regularexpression to handle months of the year spelled out, minimum and maximumnumbers may then be mapped to the regular expression multiplicity syntaxinvolving curly braces (e.g., [A-Za-z] {3, 9}).

In some embodiments, the regular expression generator 110 may keep trackof minimum and maximum number of occurrences for each span, but also mayhandle additional implementation details. For example, as a result ofthe combination of handling optional spans and running LCS on spans ofcharacters, the regular expression generator 110 may be configured todetect and merge consecutive spans throughout the execution of the LCSalgorithm. Additionally, the any extra optional spans being carriedalong sometimes appearing consecutively, and it may be desirable for theLCS algorithm to be run on those recursively as well. For example, insome cases, the regular expression generator 110 modify and/or extendthe LCS algorithm to favor (or weight) fewer transitions betweenoptional and required sequence elements (e.g., spans). For example,grouping optional spans together may minimize the number of groupingparentheses that have to be used within the regular expression, whichmay thus improve the human readability of the generated regularexpression. In some cases, if the resultant lengths are equal even afterconsidering optional spans, then the regular expression generator 110may exhibit a preference for the alternative with fewer transitionsbetween optional and required spans. For example, in some cases astandard LCS algorithm may be implemented to prefer the choice of longersequences at its decision points. However, at decision points where theoptions are of equal length, a configuration preference may beprogrammed into the regular expression generator 110. One suchconfiguration preference may be, for example, is to prefer shortersequences (once optional spans are considered). Thus, the customized LCSwithin this configuration may simultaneously optimize for longersequences (of required spans) and shorter sequences (of total requiredand optional spans).

In some embodiments, generated regular expressions may be more readableif they begin with a required span (which may also serve as a mentalanchor to a human reader), rather than starting the regular expressionswith optional spans. Thus, in some cases, if the resultant options haveequal numbers of transitions, then the option with earlier non-optionalspans may be chosen. Additionally, the LCS algorithm executed by theregular expression generator 110 may be configured in some embodimentsto push all spaces (including optional spans corresponding to spaces) tothe right within the regular expression. By pushing all the spaces tothe right, there may be an increased chance that spans of spaces may bemerged together, which may simplify the resulting regular expression aswell as improving readability. Thus, during the execution of the LCSalgorithm, when a determination is made that two sets of substrings havethe same LCS, instead of arbitrarily selecting one of the two sets, theset that facilitates improved readability may be selected. Further, insome embodiments, the LCS algorithm may be configured to favor a greaternumber of required spans, and/or fewer optional spans, in order toimprove readability.

As noted above, negative examples also may be based on optional spans insome cases. For example, the user may provide positive examples of “ab”and “a2b” and a negative example of “a3b”. In this case, an exampleimplementation may fail, because it may attempt to discriminate basedonly on required spans and the “2” digit is in an optional span. In suchcases, the user may alerted to the failure and may be provided theoptions, via the user interface, to manually repair the generatedregular expression and/or to remove some of the negative examples.

In some embodiments, there may be an isSuccess returned as part of theJSON coming back from the REST service. In some embodiments, thegenerated regex may become a different color (e.g., red) whenisSuccess=false.

VII. Regular Expression Generation Using Combinatoric Longest CommonSubsequence Algorithms

Further aspects described herein relate to a combinatoric search, inwhich the LCS algorithm executed by the regular expression generator 110may be run multiple times to generate a “correct” regular expression(e.g., a regular expression that properly matches all given positiveexamples and properly excludes all given negative examples), and/or togenerate multiple correct regular expressions from which a mostdesirable or optimal regular expression may be selected. For example,during a combinatoric search, the full LCS algorithm and regularexpression generation process may be run multiple times, includingdifferent combinations/permutations of text processing directions,different anchoring, and other different characteristics of the LCSalgorithm.

FIG. 20 is a flowchart illustrating a process for generating regularexpressions based on combinatoric executions of a longest commonsubsequence (LCS) algorithm. In step 2001, the regular expressiongenerator 110 may receive input data character sequences correspondingto positive examples. In step 2002, the regular expression generator 110may iterate over various different combinations of execution techniquesfor the LCS algorithm. As shown in this examples, during each iterationof steps 2002, the regular expression generator 110 may select adifferent combination of the following LCS algorithm executionparameters (or characteristics): anchor (i.e., no anchoring, anchoringto the beginning of the line, anchoring to the end of the line),processing direction (i.e., right-to-left order, left-to-right order),push space (i.e., do or do not push spaces), and collapse spans (i.e.,do or not collapse spans). In step 2003, the LCS algorithm is run on theinput data character sequences (or on regular expression codes if theinput character sequences were converted first), wherein the LCSalgorithm is configured based on the parameters/characteristics selectedin step 2002. In step 2004, the output of the LCS algorithm of may bestored by the regular expression generator 110, include data such aswhether or not an LCS was successfully identified by the algorithm, andthe length of the corresponding regular expression. In step 2005, theprocess may iterate until the LCS algorithm has been run with allpossible combinations of the parameters/characteristics of thecombinatoric search. Finally, in step 2006, a particular output from oneof the LCS is selected as an optimal output (e.g., based on success andregular expression length), and a regular expression may be generatedbased on the selected LCS algorithm output.

In various embodiments, a combinatoric search such as that describedabove in reference to FIG. 20 , may be performed for various differentcombinations of parameters/characteristics. For example, in someembodiments an LCS algorithm may use the caret symbol {circumflex over( )} to anchor the regular expression to the beginning of the text,and/or the dollar symbol $ to anchor the regular expression to the endof the text. In some cases, such anchoring may result in generating ashorter regular expression. Anchors may be particularly useful when auser wishes to find a particular pattern at the beginning and/or at theend of a string. For example, a user may want a product name at thebeginning. To avoid confusing the LCS algorithm with the varying numberof words describing the product name, a caret may be used to anchor theregex to the beginning of a string as depicted in the image below.

Additionally, in some embodiments, the LCS algorithm may be executedwith input data that is either forward or reversed (or similarly the LCSalgorithm may be configured to receive the input data in the usual orderand then reverse the order before executing the algorithm). Thus, insome embodiments, a combinatoric search of LCS algorithms that may beperformed on a pair of input character sequences or codes may be:

-   -   1. Usual (right-to-left) order, no anchoring to start or end    -   2. Usual (right-to-left) order, anchoring to beginning of line        using caret {circumflex over ( )}    -   3. Usual (right-to-left) order, anchoring to end of line using        dollar $    -   4. Reverse (left-to-right) order, no anchoring to start or end    -   5. Reverse (left-to-right) order, anchoring to beginning of line        using caret {circumflex over ( )}    -   6. Reverse (left-to-right) order, anchoring to end of line using        dollar $        In this example, out of the six executions of the LCS, the        shortest resulting regular expression may be selected (step        2006).

In some embodiments, the combinatoric search of the LCS algorithm alsomay iterate over a greedy quantifier “?” and non-greedy quantifier “??”.For example, by default if there is an optional span a single questionmark is emitted, e.g., [A-Z]+(?: [A-Z]\.)? [A-Z]+ for first and lastname with optional middle initial. If a satisfactory regular expressioncannot be found when using the greedy quantifier, then the combinatoricsearch may attempt to replace all question mark quantifiers withdouble-question mark quantifiers (e.g., [A-Z]+(?: [A-Z]\.)?? [A-Z]+).The double question mark corresponds to a non-greedy quantifier, whichmay instruct a downstream regular expression matcher to go intobacktracking mode in order to find a match.

Additionally, in some embodiments, the combinatoric search of the LCSalgorithm also may iterate over whether or not to prefer spaces on theright. For example, as noted above, a strategy may be used in someembodiments of pushing spaces to the right, e.g., when the LCS algorithmis faced with an arbitrary choice of otherwise equal options, in thehope that space spans may get merged together, resulting in a fewernumber of overall spans. This feature adds another option to thecombinatoric search, that is, to either push spaces to the right orexecute in accordance with a traditional LCS approach of leaving thedecision to be arbitrary.

Further, in some embodiments, the combinatoric search of LCS algorithmalso may iterative over scanning/not scanning for literals common amongall the examples, by running LCS on the original strings. In suchembodiments, the LCS algorithm may be configured to identify and alignon common words. As used herein, a “common word” may refer to a wordthat appears in every positive example. Once a common word isidentified, its span type may be converted from LETTER to WORD, and thesubsequent run through the LCS algorithm may then naturally aligns onit.

Thus, in the example below, a combinatoric search may iterate overseveral parameters/characteristics to reach 96 times that the completeLCS algorithm is to be performed. The various parameters/characteristicsto be iterated over in this example are:

-   -   Anchor (3) (Values={circumflex over ( )}, $, or neither)    -   Pushing Spaces (2) (Values=Yes or No)    -   Coalescing Low Cardinality Spans to Wildcards (2) (Values=Yes or        No)    -   Greedy Quantifier ? (2) (Values=Yes or No)    -   Aligning the LCS Algorithm on Common Tokens (2) (Values=Yes or        No)    -   Using “\w” to Represent Alphanumeric, Versus Keeping Letters        “\pL” and Numbers “\pN” Treated as Separate Spans (2)        (Values=Yes or No)        As noted above, in this example, the complete LCS algorithm is        to be performed 96 times (e.g., 3*2*2*2*2*2=96).

However, in other embodiments, the regular expression generator 110 mayprovide a performance enhancement, by which only the first threecharacteristics in the above list (Anchor, Pushing Spaces, andCoalescing Low Cardinality Spans to Wildcard, may participate in thecombinatoric search. This may result in a far fewer number of completeLCS algorithm is to be performed (e.g., 3*2*2=12 times). In suchembodiments, while the last three characteristics in the above list(Greedy Quantifier, Aligning the LCS Algorithm on Common Tokens, andUsing “\w” to Represent Alphanumeric, Versus Keeping Letters “\pL” andNumbers “\pN” Treated as Separate Spans) do not participate in thecombinatoric search, these characteristics may be tested at the end,individually and serially. Technical advantages may be realized in suchembodiments, because dividing the search space in this manner may stillresulted in a satisfactory regular expression being found, but withapproximately an 8× speedup in performance.

To illustrate, the following example of a combinatoric search mayprovide a performance advantage over the previous example. In thisexample, the combinatoric search may be performed based on the followingparameters/characteristics to be iterated over:

-   -   Anchoring (3): BEGINNING_OF_LINE_MODE, END_OF_LINE_MODE,        NO_EOL_MODE    -   Order/Direction (2): Right-to-left (normal) LCS vs.        Left-to-right (reverse) LCS    -   Push (2): Whether or not to try to push spaces to the right        within the LCS algorithm    -   Compress to Wildcards (2): Whether or not to try to compress        long sequences of only-sometimes occurring spans down to the        wildcards.*?

The combinatoric in this example may result in running the completealgorithm 3*2*2*2=24 times). The regular expression generator 110 thenmay take the best of the 24 results of the LCS algorithm, where “best”may means that (a) the LCS algorithm succeeded, and (b) the shortestregular expression was generated. The regular expression generator 110then may perform the following three additional tasks:

-   -   1. Try condensing sequences of letters and numbers that are        unbroken by spaces, punctuation, or symbols, down to a new span        type I called ALPHANUMERIC, corresponding to generated regex        of 4. This may be useful for hexadecimal numbers as found in        IPv6 addresses from clickstream logs (refer to novelty 64 from        April 2019).    -   2. Try using the non-greedy quantifier ?? instead of the greedy        quantifier ?    -   3. Try aligning on literals        VIII. Hardware Overview

FIG. 21 depicts a simplified diagram of a distributed system 2100 forimplementing an embodiment. In the illustrated embodiment, distributedsystem 2100 includes one or more client computing devices 2102, 2104,2106, and 2108, coupled to a server 2112 via one or more communicationnetworks 2110. Clients computing devices 2102, 2104, 2106, and 2108 maybe configured to execute one or more applications.

In various embodiments, server 2112 may be adapted to run one or moreservices or software applications that enable automated generation ofregular expressions, as described in this disclosure. For example, incertain embodiments, server 2112 may receive user input data transmittedfrom a client device, where the user input data is received by theclient device through a user interface displayed at the client device.Server 2112 may then convert the user input data into a regularexpression that is transmitted to the client device for display throughthe user interface.

In certain embodiments, server 2112 may also provide other services orsoftware applications that can include non-virtual and virtualenvironments. In some embodiments, these services may be offered asweb-based or cloud services, such as under a Software as a Service(SaaS) model to the users of client computing devices 2102, 2104, 2106,and/or 2108. Users operating client computing devices 2102, 2104, 2106,and/or 2108 may in turn utilize one or more client applications tointeract with server 2112 to utilize the services provided by thesecomponents.

In the configuration depicted in FIG. 21 , server 2112 may include oneor more components 2118, 2120 and 2122 that implement the functionsperformed by server 2112. These components may include softwarecomponents that may be executed by one or more processors, hardwarecomponents, or combinations thereof. It should be appreciated thatvarious different system configurations are possible, which may bedifferent from distributed system 2100. The embodiment shown in FIG. 21is thus one example of a distributed system for implementing anembodiment system and is not intended to be limiting.

Users may use client computing devices 2102, 2104, 2106, and/or 2108 toexecute one or more applications, which may generate regular expressionsin accordance with the teachings of this disclosure. A client device mayprovide an interface that enables a user of the client device tointeract with the client device. The client device may also outputinformation to the user via this interface. Although FIG. 21 depictsonly four client computing devices, any number of client computingdevices may be supported.

The client devices may include various types of computing systems suchas portable handheld devices, general purpose computers such as personalcomputers and laptops, workstation computers, wearable devices, gamingsystems, thin clients, various messaging devices, sensors or othersensing devices, and the like. These computing devices may run varioustypes and versions of software applications and operating systems (e.g.,Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operatingsystems, Linux or Linux-like operating systems such as Google Chrome™OS) including various mobile operating systems (e.g., Microsoft WindowsMobile®, iOS®, Windows Phone®, Android™, BlackBerry®, Palm OS®).Portable handheld devices may include cellular phones, smartphones,(e.g., an iPhone®), tablets (e.g., iPad®), personal digital assistants(PDAs), and the like. Wearable devices may include Google Glass® headmounted display, and other devices. Gaming systems may include varioushandheld gaming devices, Internet-enabled gaming devices (e.g., aMicrosoft Xbox® gaming console with or without a Kinect® gesture inputdevice, Sony PlayStation® system, various gaming systems provided byNintendo®, and others), and the like. The client devices may be capableof executing various different applications such as variousInternet-related apps, communication applications (e.g., E-mailapplications, short message service (SMS) applications) and may usevarious communication protocols.

Network(s) 2110 may be any type of network familiar to those skilled inthe art that can support data communications using any of a variety ofavailable protocols, including without limitation TCP/IP (transmissioncontrol protocol/Internet protocol), SNA (systems network architecture),IPX (Internet packet exchange), AppleTalk®, and the like. Merely by wayof example, network(s) 2110 can be a local area network (LAN), networksbased on Ethernet, Token-Ring, a wide-area network (WAN), the Internet,a virtual network, a virtual private network (VPN), an intranet, anextranet, a public switched telephone network (PSTN), an infra-rednetwork, a wireless network (e.g., a network operating under any of theInstitute of Electrical and Electronics (IEEE) 1002.11 suite ofprotocols, Bluetooth®, and/or any other wireless protocol), and/or anycombination of these and/or other networks.

Server 2112 may be composed of one or more general purpose computers,specialized server computers (including, by way of example, PC (personalcomputer) servers, UNIX® servers, mid-range servers, mainframecomputers, rack-mounted servers, etc.), server farms, server clusters,or any other appropriate arrangement and/or combination. Server 2112 caninclude one or more virtual machines running virtual operating systems,or other computing architectures involving virtualization such as one ormore flexible pools of logical storage devices that can be virtualizedto maintain virtual storage devices for the server. In variousembodiments, server 2112 may be adapted to run one or more services orsoftware applications that provide the functionality described in theforegoing disclosure.

The computing systems in server 2112 may run one or more operatingsystems including any of those discussed above, as well as anycommercially available server operating system. Server 2112 may also runany of a variety of additional server applications and/or mid-tierapplications, including HTTP (hypertext transport protocol) servers, FTP(file transfer protocol) servers, CGI (common gateway interface)servers, JAVA® servers, database servers, and the like. Exemplarydatabase servers include without limitation those commercially availablefrom Oracle®, Microsoft®, Sybase®, IBM® (International BusinessMachines), and the like.

In some implementations, server 2112 may include one or moreapplications to analyze and consolidate data feeds and/or event updatesreceived from users of client computing devices 2102, 2104, 2106, and2108. As an example, data feeds and/or event updates may include, butare not limited to, Twitter® feeds, Facebook® updates or real-timeupdates received from one or more third party information sources andcontinuous data streams, which may include real-time events related tosensor data applications, financial tickers, network performancemeasuring tools (e.g., network monitoring and traffic managementapplications), clickstream analysis tools, automobile trafficmonitoring, and the like. Server 2112 may also include one or moreapplications to display the data feeds and/or real-time events via oneor more display devices of client computing devices 2102, 2104, 2106,and 2108.

Distributed system 2100 may also include one or more data repositories2114, 2116. These data repositories may be used to store data and otherinformation in certain embodiments. For example, one or more of the datarepositories 2114, 2116 may be used to store information such as a newcolumn of data that matches a system-generated regular expression. Datarepositories 2114, 2116 may reside in a variety of locations. Forexample, a data repository used by server 2112 may be local to server2112 or may be remote from server 2112 and in communication with server2112 via a network-based or dedicated connection. Data repositories2114, 2116 may be of different types. In certain embodiments, a datarepository used by server 2112 may be a database, for example, arelational database, such as databases provided by Oracle Corporation®and other vendors. One or more of these databases may be adapted toenable storage, update, and retrieval of data to and from the databasein response to SQL-formatted commands.

In certain embodiments, one or more of data repositories 2114, 2116 mayalso be used by applications to store application data. The datarepositories used by applications may be of different types such as, forexample, a key-value store repository, an object store repository, or ageneral storage repository supported by a file system.

In certain embodiments, the functionalities described in this disclosuremay be offered as services via a cloud environment. FIG. 22 is asimplified block diagram of a cloud-based system environment in whichvarious services may be offered as cloud services, in accordance withcertain examples. In the example depicted in FIG. 22 , cloudinfrastructure system 2202 may provide one or more cloud services thatmay be requested by users using one or more client computing devices2204, 2206, and 2208. Cloud infrastructure system 2202 may comprise oneor more computers and/or servers that may include those described abovefor server 2112. The computers in cloud infrastructure system 2202 maybe organized as general purpose computers, specialized server computers,server farms, server clusters, or any other appropriate arrangementand/or combination.

Network(s) 2210 may facilitate communication and exchange of databetween clients 2204, 2206, and 2208 and cloud infrastructure system2202. Network(s) 2210 may include one or more networks. The networks maybe of the same or different types. Network(s) 2210 may support one ormore communication protocols, including wired and/or wireless protocols,for facilitating the communications.

The example depicted in FIG. 22 is only one example of a cloudinfrastructure system and is not intended to be limiting. It should beappreciated that, in some other examples, cloud infrastructure system2202 may have more or fewer components than those depicted in FIG. 22 ,may combine two or more components, or may have a differentconfiguration or arrangement of components. For example, although FIG.22 depicts three client computing devices, any number of clientcomputing devices may be supported in alternative examples.

The term cloud service is generally used to refer to a service that ismade available to users on demand and via a communication network suchas the Internet by systems (e.g., cloud infrastructure system 2202) of aservice provider. Typically, in a public cloud environment, servers andsystems that make up the cloud service provider's system are differentfrom the customer's own on-premise servers and systems. The cloudservice provider's systems are managed by the cloud service provider.Customers may thus avail themselves of cloud services provided by acloud service provider without having to purchase separate licenses,support, or hardware and software resources for the services. Forexample, a cloud service provider's system may host an application, anda user may, via the Internet, on demand, order and use the applicationwithout the user having to buy infrastructure resources for executingthe application. Cloud services are designed to provide easy, scalableaccess to applications, resources and services. Several providers offercloud services. For example, several cloud services are offered byOracle Corporation® of Redwood Shores, Calif., such as middlewareservices, database services, Java cloud services, and others.

In certain embodiments, cloud infrastructure system 2202 may provide oneor more cloud services using different models such as under a Softwareas a Service (SaaS) model, a Platform as a Service (PaaS) model, anInfrastructure as a Service (IaaS) model, and others, including hybridservice models. Cloud infrastructure system 2202 may include a suite ofapplications, middleware, databases, and other resources that enableprovision of the various cloud services.

A SaaS model enables an application or software to be delivered to acustomer over a communication network like the Internet, as a service,without the customer having to buy the hardware or software for theunderlying application. For example, a SaaS model may be used to providecustomers access to on-demand applications that are hosted by cloudinfrastructure system 2202. Examples of SaaS services provided by OracleCorporation® include, without limitation, various services for humanresources/capital management, customer relationship management (CRM),enterprise resource planning (ERP), supply chain management (SCM),enterprise performance management (EPM), analytics services, socialapplications, and others.

An IaaS model is generally used to provide infrastructure resources(e.g., servers, storage, hardware and networking resources) to acustomer as a cloud service to provide elastic compute and storagecapabilities. Various IaaS services are provided by Oracle Corporation®.

A PaaS model is generally used to provide, as a service, platform andenvironment resources that enable customers to develop, run, and manageapplications and services without the customer having to procure, build,or maintain such resources. Examples of PaaS services provided by OracleCorporation® include, without limitation, Oracle Java Cloud Service(JCS), Oracle Database Cloud Service (DBCS), data management cloudservice, various application development solutions services, and others.

Cloud services are generally provided on an on-demand self-servicebasis, subscription-based, elastically scalable, reliable, highlyavailable, and secure manner. For example, a customer, via asubscription order, may order one or more services provided by cloudinfrastructure system 2202. Cloud infrastructure system 2202 thenperforms processing to provide the services requested in the customer'ssubscription order. Cloud infrastructure system 2202 may be configuredto provide one or more cloud services.

Cloud infrastructure system 2202 may provide the cloud services viadifferent deployment models. In a public cloud model, cloudinfrastructure system 2202 may be owned by a third party cloud servicesprovider and the cloud services are offered to any general publiccustomer, where the customer may be an individual or an enterprise.Under a private cloud model, cloud infrastructure system 2202 may beoperated within an organization (e.g., within an enterpriseorganization) and services provided to customers that are within theorganization. For example, the customers may be various departments ofan enterprise such as the Human Resources department, the Payrolldepartment, etc. or even individuals within the enterprise. Under acommunity cloud model, the cloud infrastructure system 2202 and theservices provided may be shared by several organizations in a relatedcommunity. Various other models such as hybrids of the above mentionedmodels may also be used.

Client computing devices 2204, 2206, and 2208 may be of different types(such as devices 2102, 2104, 2106, and 2108 depicted in FIG. 21 ) andmay be capable of operating one or more client applications. A user mayuse a client device to interact with cloud infrastructure system 2202,such as to request a service provided by cloud infrastructure system2202.

In some embodiments, the processing performed by cloud infrastructuresystem 2202 for providing management-related services may involve bigdata analysis. This analysis may involve using, analyzing, andmanipulating large data sets to detect and visualize various trends,behaviors, relationships, etc. within the data. This analysis may beperformed by one or more processors, possibly processing the data inparallel, performing simulations using the data, and the like. Forexample, big data analysis may be performed by cloud infrastructuresystem 2202 for determining regular expressions in an automated manner.The data used for this analysis may include structured data (e.g., datastored in a database or structured according to a structured model)and/or unstructured data (e.g., data blobs (binary large objects)).

As depicted in the example in FIG. 22 , cloud infrastructure system 2202may include infrastructure resources 2230 that are utilized forfacilitating the provision of various cloud services offered by cloudinfrastructure system 2202. Infrastructure resources 2230 may include,for example, processing resources, storage or memory resources,networking resources, and the like.

In certain embodiments, to facilitate efficient provisioning of theseresources for supporting the various cloud services provided by cloudinfrastructure system 2202 for different customers, the resources may bebundled into sets of resources or resource modules (also referred to as“pods”). Each resource module or pod may comprise a pre-integrated andoptimized combination of resources of one or more types. In certainembodiments, different pods may be pre-provisioned for different typesof cloud services. For example, a first set of pods may be provisionedfor a database service, a second set of pods, which may include adifferent combination of resources than a pod in the first set of pods,may be provisioned for Java service, and the like. For some services,the resources allocated for provisioning the services may be sharedbetween the services.

Cloud infrastructure system 2202 may itself internally use services 2232that are shared by different components of cloud infrastructure system2202 and which facilitate the provisioning of services by cloudinfrastructure system 2202. These internal shared services may include,without limitation, a security and identity service, an integrationservice, an enterprise repository service, an enterprise managerservice, a virus scanning and white list service, a high availability,backup and recovery service, service for enabling cloud support, anemail service, a notification service, a file transfer service, and thelike.

Cloud infrastructure system 2202 may comprise multiple subsystems. Thesesubsystems may be implemented in software, or hardware, or combinationsthereof. As depicted in FIG. 22 , the subsystems may include a userinterface subsystem 2212 that enables users or customers of cloudinfrastructure system 2202 to interact with cloud infrastructure system2202. User interface subsystem 2212 may include various differentinterfaces such as a web interface 2214, an online store interface 2216where cloud services provided by cloud infrastructure system 2202 areadvertised and are purchasable by a consumer, and other interfaces 2218.For example, a customer may, using a client device, request (servicerequest 2234) one or more services provided by cloud infrastructuresystem 2202 using one or more of interfaces 2214, 2216, and 2218. Forexample, a customer may access the online store, browse cloud servicesoffered by cloud infrastructure system 2202, and place a subscriptionorder for one or more services offered by cloud infrastructure system2202 that the customer wishes to subscribe to. The service request mayinclude information identifying the customer and one or more servicesthat the customer desires to subscribe to. For example, a customer mayplace a subscription order for anautomated-generation-of-regular-expressions-related service offered bycloud infrastructure system 2202.

In certain embodiments, such as the example depicted in FIG. 22 , cloudinfrastructure system 2202 may comprise an order management subsystem(OMS) 2220 that is configured to process the new order. As part of thisprocessing, OMS 2220 may be configured to: create an account for thecustomer, if not done already; receive billing and/or accountinginformation from the customer that is to be used for billing thecustomer for providing the requested service to the customer; verify thecustomer information; upon verification, book the order for thecustomer; and orchestrate various workflows to prepare the order forprovisioning.

Once properly validated, OMS 2220 may then invoke the order provisioningsubsystem (OPS) 2224 that is configured to provision resources for theorder including processing, memory, and networking resources. Theprovisioning may include allocating resources for the order andconfiguring the resources to facilitate the service requested by thecustomer order. The manner in which resources are provisioned for anorder and the type of the provisioned resources may depend upon the typeof cloud service that has been ordered by the customer. For example,according to one workflow, OPS 2224 may be configured to determine theparticular cloud service being requested and identify a number of podsthat may have been pre-configured for that particular cloud service. Thenumber of pods that are allocated for an order may depend upon thesize/amount/level/scope of the requested service. For example, thenumber of pods to be allocated may be determined based upon the numberof users to be supported by the service, the duration of time for whichthe service is being requested, and the like. The allocated pods maythen be customized for the particular requesting customer for providingthe requested service.

Cloud infrastructure system 2202 may send a response or notification2244 to the requesting customer to indicate when the requested serviceis now ready for use. In some instances, information (e.g., a link) maybe sent to the customer that enables the customer to start using andavailing the benefits of the requested services. In certain embodiments,for a customer requesting theautomated-generation-of-regular-expressions-related service, theresponse may include instructions which, when executed, cause display ofa user interface.

Cloud infrastructure system 2202 may provide services to multiplecustomers. For each customer, cloud infrastructure system 2202 isresponsible for managing information related to one or more subscriptionorders received from the customer, maintaining customer data related tothe orders, and providing the requested services to the customer. Cloudinfrastructure system 2202 may also collect usage statistics regarding acustomer's use of subscribed services. For example, statistics may becollected for the amount of storage used, the amount of datatransferred, the number of users, and the amount of system up time andsystem down time, and the like. This usage information may be used tobill the customer. Billing may be done, for example, on a monthly cycle.

Cloud infrastructure system 2202 may provide services to multiplecustomers in parallel. Cloud infrastructure system 2202 may storeinformation for these customers, including possibly proprietaryinformation. In certain embodiments, cloud infrastructure system 2202comprises an identity management subsystem (IMS) 2228 that is configuredto manage customer information and provide the separation of the managedinformation such that information related to one customer is notaccessible by another customer. IMS 2228 may be configured to providevarious security-related services such as identity services; informationaccess management, authentication and authorization services; servicesfor managing customer identities and roles and related capabilities, andthe like.

FIG. 23 illustrates an example of computer system 2300. In someembodiments, computer system 2300 may be used to implement any of thesystems described above. As shown in FIG. 23 , computer system 2300includes various subsystems including a processing subsystem 2304 thatcommunicates with a number of other subsystems via a bus subsystem 2302.These other subsystems may include processing acceleration unit 2306,I/O subsystem 2308, storage subsystem 2318, and communications subsystem2324. Storage subsystem 2318 may include non-transitorycomputer-readable storage media including storage media 2322 and systemmemory 2310.

Bus subsystem 2302 provides a mechanism for letting the variouscomponents and subsystems of computer system 2300 communicate with eachother as intended. Although bus subsystem 2302 is shown schematically asa single bus, alternative examples of the bus subsystem may utilizemultiple buses. Bus subsystem 2302 may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, a local bus using any of a variety of bus architectures, and thelike. For example, such architectures may include an Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnect (PCI) bus, which may beimplemented as a Mezzanine bus manufactured to the IEEE P1386.1standard, and the like.

Processing subsystem 2304 controls the operation of computer system 2300and may comprise one or more processors, application specific integratedcircuits (ASICs), or field programmable gate arrays (FPGAs). Theprocessors may include be single core or multicore processors. Theprocessing resources of computer system 2300 may be organized into oneor more processing units 2332, 2334, etc. A processing unit may includeone or more processors, one or more cores from the same or differentprocessors, a combination of cores and processors, or other combinationsof cores and processors. In some embodiments, processing subsystem 2304may include one or more special purpose co-processors such as graphicsprocessors, digital signal processors (DSPs), or the like. In someembodiments, some or all of the processing units of processing subsystem2304 may be implemented using customized circuits, such as applicationspecific integrated circuits (ASICs), or field programmable gate arrays(FPGAs).

In some embodiments, the processing units in processing subsystem 2304may execute instructions stored in system memory 2310 or on computerreadable storage media 2322. In various examples, the processing unitsmay execute a variety of programs or code instructions and may maintainmultiple concurrently executing programs or processes. At any giventime, some or all of the program code to be executed may be resident insystem memory 2310 and/or on computer-readable storage media 2322including potentially on one or more storage devices. Through suitableprogramming, processing subsystem 2304 may provide variousfunctionalities described above. In instances where computer system 2300is executing one or more virtual machines, one or more processing unitsmay be allocated to each virtual machine.

In certain embodiments, a processing acceleration unit 2306 mayoptionally be provided for performing customized processing or foroff-loading some of the processing performed by processing subsystem2304 so as to accelerate the overall processing performed by computersystem 2300.

I/O subsystem 2308 may include devices and mechanisms for inputtinginformation to computer system 2300 and/or for outputting informationfrom or via computer system 2300. In general, use of the term inputdevice is intended to include all possible types of devices andmechanisms for inputting information to computer system 2300. Userinterface input devices may include, for example, a keyboard, pointingdevices such as a mouse or trackball, a touchpad or touch screenincorporated into a display, a scroll wheel, a click wheel, a dial, abutton, a switch, a keypad, audio input devices with voice commandrecognition systems, microphones, and other types of input devices. Userinterface input devices may also include motion sensing and/or gesturerecognition devices such as the Microsoft Kinect® motion sensor thatenables users to control and interact with an input device, theMicrosoft Xbox® 360 game controller, devices that provide an interfacefor receiving input using gestures and spoken commands. User interfaceinput devices may also include eye gesture recognition devices such asthe Google Glass® blink detector that detects eye activity (e.g.,“blinking” while taking pictures and/or making a menu selection) fromusers and transforms the eye gestures as inputs to an input device(e.g., Google Glass®). Additionally, user interface input devices mayinclude voice recognition sensing devices that enable users to interactwith voice recognition systems (e.g., Siri® navigator) through voicecommands.

Other examples of user interface input devices include, withoutlimitation, three dimensional (3D) mice, joysticks or pointing sticks,gamepads and graphic tablets, and audio/visual devices such as speakers,digital cameras, digital camcorders, portable media players, webcams,image scanners, fingerprint scanners, barcode reader 3D scanners, 3Dprinters, laser rangefinders, and eye gaze tracking devices.Additionally, user interface input devices may include, for example,medical imaging input devices such as computed tomography, magneticresonance imaging, position emission tomography, and medicalultrasonography devices. User interface input devices may also include,for example, audio input devices such as MIDI keyboards, digital musicalinstruments and the like.

In general, use of the term output device is intended to include allpossible types of devices and mechanisms for outputting information fromcomputer system 2300 to a user or other computer. User interface outputdevices may include a display subsystem, indicator lights, or non-visualdisplays such as audio output devices, etc. The display subsystem may bea cathode ray tube (CRT), a flat-panel device, such as that using aliquid crystal display (LCD) or plasma display, a projection device, atouch screen, and the like. For example, user interface output devicesmay include, without limitation, a variety of display devices thatvisually convey text, graphics and audio/video information such asmonitors, printers, speakers, headphones, automotive navigation systems,plotters, voice output devices, and modems.

Storage subsystem 2318 provides a repository or data store for storinginformation and data that is used by computer system 2300. Storagesubsystem 2318 provides a tangible non-transitory computer-readablestorage medium for storing the basic programming and data constructsthat provide the functionality of some examples. Storage subsystem 2318may store software (e.g., programs, code modules, instructions) thatwhen executed by processing subsystem 2304 provides the functionalitydescribed above. The software may be executed by one or more processingunits of processing subsystem 2304. Storage subsystem 2318 may alsoprovide a repository for storing data used in accordance with theteachings of this disclosure.

Storage subsystem 2318 may include one or more non-transitory memorydevices, including volatile and non-volatile memory devices. As shown inFIG. 23 , storage subsystem 2318 includes system memory 2310 andcomputer-readable storage media 2322. System memory 2310 may include anumber of memories including a volatile main random access memory (RAM)for storage of instructions and data during program execution and anon-volatile read only memory (ROM) or flash memory in which fixedinstructions are stored. In some implementations, a basic input/outputsystem (BIOS), containing the basic routines that help to transferinformation between elements within computer system 2300, such as duringstart-up, may typically be stored in the ROM. The RAM typically containsdata and/or program modules that are presently being operated andexecuted by processing subsystem 2304. In some implementations, systemmemory 2310 may include multiple different types of memory, such asstatic random access memory (SRAM), dynamic random access memory (DRAM),and the like.

By way of example, and not limitation, as depicted in FIG. 23 , systemmemory 2310 may load application programs 2312 that are being executed,which may include various applications such as Web browsers, mid-tierapplications, relational database management systems (RDBMS), etc.,program data 2314, and operating system 2316. By way of example,operating system 2316 may include various versions of MicrosoftWindows®, Apple Macintosh®, and/or Linux operating systems, a variety ofcommercially-available UNIX® or UNIX-like operating systems (includingwithout limitation the variety of GNU/Linux operating systems, theGoogle Chrome® OS, and the like) and/or mobile operating systems such asiOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operatingsystems, and others.

Computer-readable storage media 2322 may store programming and dataconstructs that provide the functionality of some examples.Computer-readable media 2322 may provide storage of computer-readableinstructions, data structures, program modules, and other data forcomputer system 2300. Software (programs, code modules, instructions)that, when executed by processing subsystem 2304 provides thefunctionality described above, may be stored in storage subsystem 2318.By way of example, computer-readable storage media 2322 may includenon-volatile memory such as a hard disk drive, a magnetic disk drive, anoptical disk drive such as a CD ROM, DVD, a Blu-Ray® disk, or otheroptical media. Computer-readable storage media 2322 may include, but isnot limited to, Zip® drives, flash memory cards, universal serial bus(USB) flash drives, secure digital (SD) cards, DVD disks, digital videotape, and the like. Computer-readable storage media 2322 may alsoinclude, solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.

In certain embodiments, storage subsystem 2318 may also includecomputer-readable storage media reader 2320 that may further beconnected to computer-readable storage media 2322. Reader 2320 mayreceive and be configured to read data from a memory device such as adisk, a flash drive, etc.

In certain embodiments, computer system 2300 may support virtualizationtechnologies, including but not limited to virtualization of processingand memory resources. For example, computer system 2300 may providesupport for executing one or more virtual machines. In certainembodiments, computer system 2300 may execute a program such as ahypervisor that facilitated the configuring and managing of the virtualmachines. Each virtual machine may be allocated memory, compute (e.g.,processors, cores), I/O, and networking resources. Each virtual machinegenerally runs independently of the other virtual machines. A virtualmachine typically runs its own operating system, which may be the sameas or different from the operating systems executed by other virtualmachines executed by computer system 2300. Accordingly, multipleoperating systems may potentially be run concurrently by computer system2300.

Communications subsystem 2324 provides an interface to other computersystems and networks. Communications subsystem 2324 serves as aninterface for receiving data from and transmitting data to other systemsfrom computer system 2300. For example, communications subsystem 2324may enable computer system 2300 to establish a communication channel toone or more client devices via the Internet for receiving and sendinginformation from and to the client devices.

Communication subsystem 2324 may support both wired and/or wirelesscommunication protocols. In certain embodiments, communicationssubsystem 2324 may include radio frequency (RF) transceiver componentsfor accessing wireless voice and/or data networks (e.g., using cellulartelephone technology, advanced data network technology, such as 3G, 4Gor EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.XXfamily standards, or other mobile communication technologies, or anycombination thereof), global positioning system (GPS) receivercomponents, and/or other components. In some embodiments, communicationssubsystem 2324 may provide wired network connectivity (e.g., Ethernet)in addition to or instead of a wireless interface.

Communication subsystem 2324 may receive and transmit data in variousforms. In some embodiments, in addition to other forms, communicationssubsystem 2324 may receive input communications in the form ofstructured and/or unstructured data feeds 2326, event streams 2328,event updates 2330, and the like. For example, communications subsystem2324 may be configured to receive (or send) data feeds 2326 in real-timefrom users of social media networks and/or other communication servicessuch as Twitter® feeds, Facebook® updates, web feeds such as Rich SiteSummary (RSS) feeds, and/or real-time updates from one or more thirdparty information sources.

In certain embodiments, communications subsystem 2324 may be configuredto receive data in the form of continuous data streams, which mayinclude event streams 2328 of real-time events and/or event updates2330, that may be continuous or unbounded in nature with no explicitend. Examples of applications that generate continuous data may include,for example, sensor data applications, financial tickers, networkperformance measuring tools (e.g. network monitoring and trafficmanagement applications), clickstream analysis tools, automobile trafficmonitoring, and the like.

Communications subsystem 2324 may also be configured to communicate datafrom computer system 2300 to other computer systems or networks. Thedata may be communicated in various different forms such as structuredand/or unstructured data feeds 2326, event streams 2328, event updates2330, and the like to one or more databases that may be in communicationwith one or more streaming data source computers coupled to computersystem 2300.

Computer system 2300 may be one of various types, including a handheldportable device (e.g., an iPhone® cellular phone, an iPad® computingtablet, a PDA), a wearable device (e.g., a Google Glass® head mounteddisplay), a personal computer, a workstation, a mainframe, a kiosk, aserver rack, or any other data processing system. Due to theever-changing nature of computers and networks, the description ofcomputer system 2300 depicted in FIG. 23 is intended only as a specificexample. Many other configurations having more or fewer components thanthe system depicted in FIG. 23 are possible. Based on the disclosure andteachings provided herein, a person of ordinary skill in the art willappreciate other ways and/or methods to implement the various examples.

Although specific examples have been described, various modifications,alterations, alternative constructions, and equivalents are possible.Examples are not restricted to operation within certain specific dataprocessing environments, but are free to operate within a plurality ofdata processing environments. Additionally, although certain exampleshave been described using a particular series of transactions and steps,it should be apparent to those skilled in the art that this is notintended to be limiting. Although some flowcharts describe operations asa sequential process, many of the operations may be performed inparallel or concurrently. In addition, the order of the operations maybe rearranged. A process may have additional steps not included in thefigure. Various features and aspects of the above-described examples maybe used individually or jointly.

Further, while certain examples have been described using a particularcombination of hardware and software, it should be recognized that othercombinations of hardware and software are also possible. Certainexamples may be implemented only in hardware, or only in software, orusing combinations thereof. The various processes described herein maybe implemented on the same processor or different processors in anycombination.

Where devices, systems, components or modules are described as beingconfigured to perform certain operations or functions, suchconfiguration may be accomplished, for example, by designing electroniccircuits to perform the operation, by programming programmableelectronic circuits (such as microprocessors) to perform the operationsuch as by executing computer instructions or code, or processors orcores programmed to execute code or instructions stored on anon-transitory memory medium, or any combination thereof. Processes maycommunicate using a variety of techniques including but not limited toconventional techniques for inter-process communications, and differentpairs of processes may use different techniques, or the same pair ofprocesses may use different techniques at different times.

Specific details are given in this disclosure to provide a thoroughunderstanding of the examples. However, examples may be practicedwithout these specific details. For example, well-known circuits,processes, algorithms, structures, and techniques have been shownwithout unnecessary detail in order to avoid obscuring the examples.This description provides example examples only, and is not intended tolimit the scope, applicability, or configuration of other examples.Rather, the preceding description of the examples will provide thoseskilled in the art with an enabling description for implementing variousexamples. Various changes may be made in the function and arrangement ofelements.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that additions, subtractions, deletions, and other modificationsand changes may be made thereunto without departing from the broaderspirit and scope as set forth in the claims. Thus, although specificexamples have been described, these are not intended to be limiting.Various modifications and equivalents are within the scope of thefollowing claims.

In the foregoing specification, aspects of the disclosure are describedwith reference to specific examples thereof, but those skilled in theart will recognize that the disclosure is not limited thereto. Variousfeatures and aspects of the above-described disclosure may be usedindividually or jointly. Further, examples may be utilized in any numberof environments and applications beyond those described herein withoutdeparting from the broader spirit and scope of the specification. Thespecification and drawings are, accordingly, to be regarded asillustrative rather than restrictive.

In the foregoing description, for the purposes of illustration, methodswere described in a particular order. It should be appreciated that inalternate examples, the methods may be performed in a different orderthan that described. It should also be appreciated that the methodsdescribed above may be performed by hardware components or may beembodied in sequences of machine-executable instructions, which may beused to cause a machine, such as a general-purpose or special-purposeprocessor or logic circuits programmed with the instructions to performthe methods. These machine-executable instructions may be stored on oneor more machine readable mediums, such as CD-ROMs or other type ofoptical disks, floppy diskettes, ROMs, RAMs, EPROMs, EEPROMs, magneticor optical cards, flash memory, or other types of machine-readablemediums suitable for storing electronic instructions. Alternatively, themethods may be performed by a combination of hardware and software.

Where components are described as being configured to perform certainoperations, such configuration may be accomplished, for example, bydesigning electronic circuits or other hardware to perform theoperation, by programming programmable electronic circuits (e.g.,microprocessors, or other suitable electronic circuits) to perform theoperation, or any combination thereof.

While illustrative examples of the application have been described indetail herein, it is to be understood that the inventive concepts may beotherwise variously embodied and employed, and that the appended claimsare intended to be construed to include such variations, except aslimited by the prior art.

Where components are described as being “configured to” perform certainoperations, such configuration may be accomplished, for example, bydesigning electronic circuits or other hardware to perform theoperation, by programming programmable electronic circuits (e.g.,microprocessors, or other suitable electronic circuits) to perform theoperation, or any combination thereof.

What is claimed is:
 1. A method of generating regular expressions,comprising: receiving, by a regular expression generator comprising oneor more processors, first input data comprising one or more positivecharacter sequences in a first field of a data structure, each of theone or more positive character sequences corresponding to a positiveexample that is to be matched by a regular expression generated by theregular expression generator; generating, by the regular expressiongenerator, a first regular expression, wherein the first regularexpression matches each of the one or more positive examples;displaying, by the regular expression generator, the generated firstregular expression that matches each of the one or more positiveexamples; receiving, by the regular expression generator, second inputdata comprising one or more negative character sequences in a secondfield of the data structure, each of the one or more negative charactersequences corresponding to a negative example that is not to be matchedby the regular expression generated by the regular expression generator;in response to receiving the second input data, determining whether eachof the one or more negative examples matches the first regularexpression that is generated based on the one or more positive examples;and in response to determining that at least one of the negativeexamples matches the first regular expression: (a) determining asubsequence of characters at a disambiguation location within the firstregular expression to replace; (b) determining a replacement charactersequence, wherein the replacement character sequence replaces thesubsequence of characters at the location within the first regularexpression in order to distinguish the one or more positive examplesfrom the one more negative examples in a regular expression; and (c)updating the first regular expression by replacing the determinedsubsequence of characters within the first regular expression with thereplacement character sequence; and displaying, by the regularexpression generator, the updated first regular expression.
 2. Themethod of claim 1, wherein determining the subsequence of characters atthe location within the first regular expression comprises: determiningthe location within the first regular expression; retrieving a textfragment from each of the one or more positive examples and from each ofthe one or more negative examples corresponding to the location withinthe first regular expression; and determining the subsequence ofcharacters as one or more characters at the location within the firstregular expression, from which the one or more positive examples aredistinguishable from the one more negative examples.
 3. The method ofclaim 2, wherein determining the location within the first regularexpression comprises: determining a first number of characters at aprefix portion of the first regular expression from which the one ormore positive examples are distinguishable from the one more negativeexamples; determining a second number of characters at a suffix portionof the first regular expression from which the one or more positiveexamples are distinguishable from the one more negative examples; andselecting either the prefix portion or the suffix portion as thelocation within the first regular expression, based at least in part onwhether the first number of characters or the second number ofcharacters is shorter.
 4. The method of claim 3, wherein determining thelocation within the first regular expression further comprises:executing a formula to determine the location within the first regularexpression, wherein the formula weights the prefix portion over thesuffix portion of the first regular expression.
 5. The method of claim2, wherein the determined location within the first regular expressionis a mid-span location that does not correspond to a prefix portion ofthe first regular expression or a suffix portion of the first regularexpression.
 6. The method of claim 2, wherein determining thereplacement character sequence comprises determining a plurality ofreplacement character sequences, and wherein updating the first regularexpression comprises replacing the determined subsequence of characterswithin the first regular expression with the plurality of replacementcharacter sequences.
 7. The method of claim 1, wherein determining thereplacement character sequence comprises: determining a first number ofcharacters at the location within the first regular expression, and acorresponding first number of replacement character sequences eachhaving the first number of characters, from which the one or morepositive examples are distinguishable from the one more negativeexamples; determining a second number of characters at the locationwithin the first regular expression, and a corresponding second numberof replacement character sequences each having the second number ofcharacters, from which the one or more positive examples aredistinguishable from the one more negative examples; and selectingeither the first number of characters or the second number of charactersfor the replacement character sequence within the first regularexpression, based on (a) sizes of the first and second numbers ofcharacters, and (b) sizes of the corresponding first and second numbersof replacement character sequences.
 8. A system for generating regularexpressions, the system comprising: a processing unit comprising one ormore processors; and memory storing instructions that, when executed bythe processing unit, cause the system to: receive first input datacomprising one or more positive character sequences in a first field ofa data structure, each of the one or more positive character sequencescorresponding to a positive example that is to be matched by a regularexpression generated by a regular expression generator; generate a firstregular expression, wherein the first regular expression matches each ofthe one or more positive examples; displaying, by the regular expressiongenerator, the generated first regular expression that matches each ofthe one or more positive examples; receive second input data comprisingone or more negative character sequences in a second field of the datastructure, each of the one or more negative character sequencescorresponding to a negative example that is not to be matched by theregular expression generated by the regular expression generator; inresponse to receiving the second input data, determine whether each ofthe one or more negative examples matches the first regular expressionthat is generated based on the one or more positive examples; and inresponse to determining that at least one of the negative examplesmatches the first regular expression: (a) determine a subsequence ofcharacters at a disambiguation location within the first regularexpression to replace; (b) determine a replacement character sequence,wherein the replacement character sequence replaces the subsequence ofcharacters at the location within the first regular expression in orderto distinguish the one or more positive examples from the one morenegative examples in a regular expression; and (c) update the firstregular expression by replacing the determined subsequence of characterswithin the first regular expression with the replacement charactersequence; and displaying, by the regular expression generator, theupdated first regular expression.
 9. The system of claim 8, whereindetermining the subsequence of characters at the location within thefirst regular expression comprises: determining the location within thefirst regular expression; retrieving a text fragment from each of theone or more positive examples and from each of the one or more negativeexamples corresponding to the location within the first regularexpression; and determining the subsequence of characters as one or morecharacters at the location within the first regular expression, fromwhich the one or more positive examples are distinguishable from the onemore negative examples.
 10. The system of claim 9, wherein determiningthe location within the first regular expression comprises: determininga first number of characters at a prefix portion of the first regularexpression from which the one or more positive examples aredistinguishable from the one more negative examples; determining asecond number of characters at a suffix portion of the first regularexpression from which the one or more positive examples aredistinguishable from the one more negative examples; and selectingeither the prefix portion or the suffix portion as the location withinthe first regular expression, based at least in part on whether thefirst number of characters or the second number of characters isshorter.
 11. The system of claim 10, wherein determining the locationwithin the first regular expression further comprises: executing aformula to determine the location within the first regular expression,wherein the formula weights the prefix portion over the suffix portionof the first regular expression.
 12. The system of claim 9, wherein thedetermined location within the first regular expression is a mid-spanlocation that does not correspond to a prefix portion of the firstregular expression or a suffix portion of the first regular expression.13. The system of claim 9, wherein determining the replacement charactersequence comprises determining a plurality of replacement charactersequences, and wherein updating the first regular expression comprisesreplacing the determined subsequence of characters within the firstregular expression with the plurality of replacement charactersequences.
 14. The system of claim 8, wherein determining thereplacement character sequence comprises: determining a first number ofcharacters at the location within the first regular expression, and acorresponding first number of replacement character sequences eachhaving the first number of characters, from which the one or morepositive examples are distinguishable from the one more negativeexamples; determining a second number of characters at the locationwithin the first regular expression, and a corresponding second numberof replacement character sequences each having the second number ofcharacters, from which the one or more positive examples aredistinguishable from the one more negative examples; and selectingeither the first number of characters or the second number of charactersfor the replacement character sequence within the first regularexpression, based on (a) sizes of the first and second numbers ofcharacters, and (b) sizes of the corresponding first and second numbersof replacement character sequences.
 15. A non-transitorycomputer-readable media for generating regular expressions, thecomputer-readable media comprising computer-executable instructionswhich when executed on a computer system, cause the computer system to:receive first input data comprising one or more positive charactersequences in a first field of a data structure, each of the one or morepositive character sequences corresponding to a positive example that isto be matched by a regular expression generated by a regular expressiongenerator; generate a first regular expression, wherein the firstregular expression matches each of the one or more positive examples;displaying, by the regular expression generator, the generated firstregular expression that matches each of the one or more positiveexamples; receive second input data comprising one or more negativecharacter sequences in a second field of the data structure, each of theone or more negative character sequences corresponding to a negativeexample that is not to be matched by the regular expression generated bythe regular expression generator; in response to receiving the secondinput data, determine whether each of the one or more negative examplesmatches the first regular expression that is generated based on the oneor more positive examples; and in response to determining that at leastone of the negative examples matches the first regular expression: (a)determine a subsequence of characters at a disambiguation locationwithin the first regular expression to replace; (b) determine areplacement character sequence, wherein the replacement charactersequence replaces the subsequence of characters at the location withinthe first regular expression in order to distinguish the one or morepositive examples from the one more negative examples in a regularexpression; and (c) update the first regular expression by replacing thedetermined subsequence of characters within the first regular expressionwith the replacement character sequence; and displaying, by the regularexpression generator, the updated first regular expression.
 16. Thecomputer-readable media of claim 15, wherein determining the subsequenceof characters at the location within the first regular expressioncomprises: determining the location within the first regular expression;retrieving a text fragment from each of the one or more positiveexamples and from each of the one or more negative examplescorresponding to the location within the first regular expression; anddetermining the subsequence of characters as one or more characters atthe location within the first regular expression, from which the one ormore positive examples are distinguishable from the one more negativeexamples.
 17. The computer-readable media of claim 16, whereindetermining the location within the first regular expression comprises:determining a first number of characters at a prefix portion of thefirst regular expression from which the one or more positive examplesare distinguishable from the one more negative examples; determining asecond number of characters at a suffix portion of the first regularexpression from which the one or more positive examples aredistinguishable from the one more negative examples; and selectingeither the prefix portion or the suffix portion as the location withinthe first regular expression, based at least in part on whether thefirst number of characters or the second number of characters isshorter.
 18. The computer-readable media of claim 17, whereindetermining the location within the first regular expression furthercomprises: executing a formula to determine the location within thefirst regular expression, wherein the formula weights the prefix portionover the suffix portion of the first regular expression.
 19. Thecomputer-readable media of claim 16, wherein the determined locationwithin the first regular expression is a mid-span location that does notcorrespond to a prefix portion of the first regular expression or asuffix portion of the first regular expression.
 20. Thecomputer-readable media of claim 16, wherein determining the replacementcharacter sequence comprises determining a plurality of replacementcharacter sequences, and wherein updating the first regular expressioncomprises replacing the determined subsequence of characters within thefirst regular expression with the plurality of replacement charactersequences.